[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866261#comment-15866261
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1505


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.2.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866259#comment-15866259
 ] 

ASF subversion and git services commented on NIFI-2565:
---

Commit ca887308af80ac34b756e1d2655c2a95fafeaf94 in nifi's branch 
refs/heads/master from Andre F de Miranda
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ca88730 ]

NIFI-2565 - Add Elastic copyright to notice due to the use of grok patterns 
during test

This closes #1505.

Signed-off-by: Aldrin Piri 


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866254#comment-15866254
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user apiri commented on the issue:

https://github.com/apache/nifi/pull/1505
  
@trixpan Thanks for fixing up.  Looking through the linked source for the 
logstash items, things look good.  Will get merged.


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866055#comment-15866055
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1505
  
@apiri hopefully this time it looks a bit better... yes... it is a fork of 
a fork. :crying_cat_face: 


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865816#comment-15865816
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1505
  
Epic. Will adjust


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865522#comment-15865522
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1108


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865520#comment-15865520
 ] 

ASF subversion and git services commented on NIFI-2565:
---

Commit 2ef7c15b5d300e8c5932b7b03800fcbd969aff3b in nifi's branch 
refs/heads/master from [~snamsi]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=2ef7c15 ]

NIFI-2565: add Grok parser

This closes #1108.

Signed-off-by: Andre F de Miranda 


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854174#comment-15854174
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1108
  
@joewitt when you have time can you have a final look on this one? LGTM and 
I am happy to address the two remaining cosmetic comments as part of a separate 
PR.




> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824025#comment-15824025
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r96236556
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import io.thekraken.grok.api.Grok;
+import io.thekraken.grok.api.Match;
+import io.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.StopWatch;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+
+@Tags({"Grok Processor", "grok", "log", "text", "parse", "delimit", 
"extract"})
+@CapabilityDescription("Evaluates one or more Grok Expressions against the 
content of a FlowFile, " +
+"adding the results as attributes or replacing the content of the 
FlowFile with a JSON " +
+"notation of the matched content")
+@WritesAttributes({
+@WritesAttribute(attribute = "grok.XXX", description = "Each of 
the Grok identifier that is matched in the flowfile will be added as an 
attribute, prefixed with \"grok.\" For example," +
--- End diff --

Isn't this just applicable if using flowfile-attribute as destination?


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746971#comment-15746971
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user selim-namsi commented on the issue:

https://github.com/apache/nifi/pull/1108
  
@trixpan @joewitt Sorry for the long delay, I applied the changes that you 
suggested such as, adding the custom validator in the processor class, use the 
new version of grok and removing the license section from the assembly license


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1561#comment-1561
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r87978657
  
--- Diff: nifi-commons/nifi-processor-utilities/pom.xml ---
@@ -45,5 +45,16 @@
 nifi-ssl-context-service-api
 provided
 
+
+io.thekraken
+grok
+0.1.4
--- End diff --

A new version has been released today and contains important fixes (reduced 
depencies, better feature parity with logstash, etc). May I  suggest we upgrade 
the  dependency? 


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665798#comment-15665798
 ] 

Joseph Witt commented on NIFI-2565:
---

removed the fix version for now.  Once this is ready for merge the fix version 
can be assigned.

> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665797#comment-15665797
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r87937612
  
--- Diff: nifi-assembly/LICENSE ---
@@ -1729,4 +1729,20 @@ This product bundles 'jbzip2' which is available 
under an MIT license.
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 
ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
IN
-THE SOFTWARE.
\ No newline at end of file
+THE SOFTWARE.
--- End diff --

This whole license section can be removed.  This is the assembly license 
which is to cover all binary artifacts and source in the build of nifi itself.  
The dependency of java-grok is binary only (not source) and is ASLv2 so nothing 
needs to be in this license for it.  There should be an entry for this in the 
notice similar to the many ASLv2 examples in there.   The only thing needing 
mentioned then is the copyright line from the project's license file 
https://github.com/thekrakken/java-grok/blob/master/LICENSE.  Also, this 
nifi-asembly/NOTICE change needed will also need to be in the NOTICE of the 
nifi-standard-nar as well.

Lots of words above but the short version is "No license change needed.  
Just add a small section to the nar NOTICE and assembly NOTICE to reflect this 
ASLv2 dependency specifically because it has a copyright reference in the 
license."


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665264#comment-15665264
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r87909770
  
--- Diff: 
nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/StandardValidators.java
 ---
@@ -26,6 +26,8 @@
 import java.util.concurrent.TimeUnit;
 import java.util.regex.Pattern;
 
+import oi.thekraken.grok.api.Grok;
--- End diff --

This validation roytine should not be added to standard validators in order 
to avoud impirting grok into the standard validator


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651457#comment-15651457
 ] 

Joseph Witt commented on NIFI-2565:
---

[~mattyb149] [~trixpan] [~markap14] can we get this across the finish line for 
1.1 or can we remove the fix version so it can be included when ready in 
whatever release is appropriate?  

> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608695#comment-15608695
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/1108
  
Looks like there are new merge issues, do you mind rebasing against the 
latest master?


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15606828#comment-15606828
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user selim-namsi commented on the issue:

https://github.com/apache/nifi/pull/1108
  
@trixpan @markap14 I pushed the new changes.
Could you please check the changes ?

Thanks!


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595217#comment-15595217
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1108
  
@selim-namsi - good work. We are getting there.


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595213#comment-15595213
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r84483537
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
+@SeeAlso({})
+@ReadsAttributes({@ReadsAttribute(attribute="", description="")})
+@WritesAttributes({@WritesAttribute(attribute="", description="")})
+public class GrokParser extends AbstractProcessor {
--- End diff --

So the pattern of naming is 'Verb Subject'.  It appears the point of this 
processor, from a users point of view (not the developers), is to evaluate Grok 
expressions against flow file content to replace that content with the result 
or to update a flow file attribute with that result.  If that is the case we 
could take the approach of 'EvaluateGrok' or 'GrokEvaluateText' or 
'ExtractGrok' is also fair game I think.


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595198#comment-15595198
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r84481939
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
+@SeeAlso({})
+@ReadsAttributes({@ReadsAttribute(attribute="", description="")})
+@WritesAttributes({@WritesAttribute(attribute="", description="")})
+public class GrokParser extends AbstractProcessor {
--- End diff --

To this point, similar nomenclature has been used in other places:


https://github.com/DhruvKumar/nifi-grok-processor-bundle/tree/master/nifi-grok-processors/src/main/java/dhruv/nifi/processors


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595158#comment-15595158
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r84477121
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
--- End diff --

@selim-namsi 

Perhaps you can use the description used as part of my WIP.

> "Evaluates one or more Grok Expressions against the content of a 
FlowFile, adding the results as attributes or replacing the content of the 
FlowFile with a JSON notation of the matched content"

?


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595148#comment-15595148
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r84476594
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
+@SeeAlso({})
+@ReadsAttributes({@ReadsAttribute(attribute="", description="")})
+@WritesAttributes({@WritesAttribute(attribute="", description="")})
+public class GrokParser extends AbstractProcessor {
+
+
+public static final String DESTINATION_ATTRIBUTE = 
"flowfile-attribute";
--- End diff --

@markap14 - "flowfile-attribute", "flowfile-content" is an established 
pattern:

https://github.com/apache/nifi/search?utf8=%E2%9C%93=flowfile-attribute

I am happy to address the terminology on other processors but I suggest we 
use the same naming, unless planning to change across the other processors.


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595139#comment-15595139
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r84476015
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
+@SeeAlso({})
+@ReadsAttributes({@ReadsAttribute(attribute="", description="")})
+@WritesAttributes({@WritesAttribute(attribute="", description="")})
+public class GrokParser extends AbstractProcessor {
--- End diff --

@markap14 - this is not a parser but an extractor (Grok is a hyper regex) 
so I suggest the name to be ExtractGrok (after ExtractText)


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583398#comment-15583398
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user selim-namsi commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83729426
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGrokParser.java
 ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+/**
+ * Created by snamsi on 05/10/16.
+ */
+public class TestGrokParser {
+
+private TestRunner testRunner;
+final static Path GROK_LOG_INPUT = 
Paths.get("src/test/resources/TestGrokParser/apache.log");
+final static Path GROK_TEXT_INPUT = 
Paths.get("src/test/resources/TestGrokParser/simple_text.log");
+
+
+@Before
+public void init() {
+testRunner = TestRunners.newTestRunner(GrokParser.class);
+}
+
+@Test
+public void testGrokParserWithMatchedContent() throws IOException {
+
+
+testRunner.setProperty(GrokParser.GROK_EXPRESSION, 
"%{COMMONAPACHELOG}");
+testRunner.setProperty(GrokParser.GROK_PATTERN_FILE, 
"src/test/resources/TestGrokParser/patterns");
+testRunner.enqueue(GROK_LOG_INPUT);
+testRunner.run();
+testRunner.assertAllFlowFilesTransferred(GrokParser.REL_MATCH);
+final MockFlowFile matched = 
testRunner.getFlowFilesForRelationship(GrokParser.REL_MATCH).get(0);
+
+matched.assertAttributeEquals("verb","GET");
+matched.assertAttributeEquals("response","401");
+matched.assertAttributeEquals("bytes","12846");
+matched.assertAttributeEquals("clientip","64.242.88.10");
+matched.assertAttributeEquals("auth","-");
+matched.assertAttributeEquals("timestamp","07/Mar/2004:16:05:49 
-0800");
+
matched.assertAttributeEquals("request","/twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables");
+matched.assertAttributeEquals("httpversion","1.1");
+
+}
+
+@Test
+public void testGrokParserWithUnMatchedContent() throws IOException {
+
+
+testRunner.setProperty(GrokParser.GROK_EXPRESSION, "%{ADDRESS}");
+testRunner.setProperty(GrokParser.GROK_PATTERN_FILE, 
"src/test/resources/TestGrokParser/patterns");
+testRunner.enqueue(GROK_TEXT_INPUT);
+testRunner.run();
+testRunner.assertAllFlowFilesTransferred(GrokParser.REL_NO_MATCH);
+final MockFlowFile notMatched = 
testRunner.getFlowFilesForRelationship(GrokParser.REL_NO_MATCH).get(0);
+notMatched.assertContentEquals(GROK_TEXT_INPUT);
+
+}
+
+@Test(expected = java.lang.AssertionError.class)
+public void testGrokParserWithNotFoundPatternFile() throws IOException 
{
+
+testRunner.setProperty(GrokParser.GROK_EXPRESSION, 
"%{COMMONAPACHELOG}");
+testRunner.setProperty(GrokParser.GROK_PATTERN_FILE, 
"src/test/resources/TestGrokParser/toto_file");
+testRunner.enqueue(GROK_LOG_INPUT);
+testRunner.run();
+
+}
+
+
+@Test(expected = java.lang.AssertionError.class)
--- End diff --

 For this method "testGrokParserWithBadGrokExpression", although the 
processor is throwing GrokException, when I use assertNotValid, the test fails 
with the following message "java.lang.AssertionError: Processor appears to be 
valid but expected it to be invalid"


> NiFi processor 

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583275#comment-15583275
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user selim-namsi commented on the issue:

https://github.com/apache/nifi/pull/1108
  
@markap14 Thanks for all this suggestions, I'll update the code ASAP and 
push the changes!


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583267#comment-15583267
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user selim-namsi commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83720911
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
+@SeeAlso({})
+@ReadsAttributes({@ReadsAttribute(attribute="", description="")})
+@WritesAttributes({@WritesAttribute(attribute="", description="")})
+public class GrokParser extends AbstractProcessor {
+
+
+public static final String DESTINATION_ATTRIBUTE = 
"flowfile-attribute";
+public static final String DESTINATION_CONTENT = "flowfile-content";
+private static final String APPLICATION_JSON = "application/json";
+
+public static final PropertyDescriptor GROK_EXPRESSION = new 
PropertyDescriptor
+.Builder().name("Grok Expression")
+.description("Grok expression")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor GROK_PATTERN_FILE = new 
PropertyDescriptor
+.Builder().name("Grok Pattern file")
+.description("Grok Pattern file definition")
+.required(false)
--- End diff --

@markap14  In the first version of the code, I was loading few useful 
pattern files by default, so the user's custom pattern file was not required, 
but after removing that part I forgot to update the required attribute, I'll 
fix it


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
>   

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583263#comment-15583263
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user selim-namsi commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83720540
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
+@SeeAlso({})
+@ReadsAttributes({@ReadsAttribute(attribute="", description="")})
+@WritesAttributes({@WritesAttribute(attribute="", description="")})
+public class GrokParser extends AbstractProcessor {
+
+
+public static final String DESTINATION_ATTRIBUTE = 
"flowfile-attribute";
+public static final String DESTINATION_CONTENT = "flowfile-content";
+private static final String APPLICATION_JSON = "application/json";
+
+public static final PropertyDescriptor GROK_EXPRESSION = new 
PropertyDescriptor
+.Builder().name("Grok Expression")
+.description("Grok expression")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor GROK_PATTERN_FILE = new 
PropertyDescriptor
+.Builder().name("Grok Pattern file")
+.description("Grok Pattern file definition")
+.required(false)
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DESTINATION = new 
PropertyDescriptor.Builder()
+.name("Destination")
+.description("Control if Grok output value is written as a new 
flowfile attribute  " +
--- End diff --

@markap14  Actually what I meant 

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572100#comment-15572100
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83229088
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGrokParser.java
 ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+/**
+ * Created by snamsi on 05/10/16.
--- End diff --

We should not have usernames here, as Git will provide this information for 
us.


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572099#comment-15572099
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83229023
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
+@SeeAlso({})
+@ReadsAttributes({@ReadsAttribute(attribute="", description="")})
+@WritesAttributes({@WritesAttribute(attribute="", description="")})
+public class GrokParser extends AbstractProcessor {
+
+
+public static final String DESTINATION_ATTRIBUTE = 
"flowfile-attribute";
+public static final String DESTINATION_CONTENT = "flowfile-content";
+private static final String APPLICATION_JSON = "application/json";
+
+public static final PropertyDescriptor GROK_EXPRESSION = new 
PropertyDescriptor
+.Builder().name("Grok Expression")
+.description("Grok expression")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor GROK_PATTERN_FILE = new 
PropertyDescriptor
+.Builder().name("Grok Pattern file")
+.description("Grok Pattern file definition")
+.required(false)
+.addValidator(StandardValidators.FILE_EXISTS_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DESTINATION = new 
PropertyDescriptor.Builder()
+.name("Destination")
+.description("Control if Grok output value is written as a new 
flowfile attribute  " +
+"or written in the flowfile content. 

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572089#comment-15572089
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83226835
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
--- End diff --

We should consider several more tags: grok, log, text, parse, delimit, 
extract


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572094#comment-15572094
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83228313
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
+@SeeAlso({})
+@ReadsAttributes({@ReadsAttribute(attribute="", description="")})
+@WritesAttributes({@WritesAttribute(attribute="", description="")})
+public class GrokParser extends AbstractProcessor {
+
+
+public static final String DESTINATION_ATTRIBUTE = 
"flowfile-attribute";
+public static final String DESTINATION_CONTENT = "flowfile-content";
+private static final String APPLICATION_JSON = "application/json";
+
+public static final PropertyDescriptor GROK_EXPRESSION = new 
PropertyDescriptor
+.Builder().name("Grok Expression")
+.description("Grok expression")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
--- End diff --

We should probably use a custom validator to make sure that the configured 
value is valid.


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was 

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572090#comment-15572090
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83226928
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
+@SeeAlso({})
--- End diff --

No need for the @SeeAlso, @ReadsAtributes, and @WritesAttributes 
annotations if they are not being used.


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572088#comment-15572088
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83226308
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
+@SeeAlso({})
+@ReadsAttributes({@ReadsAttribute(attribute="", description="")})
+@WritesAttributes({@WritesAttribute(attribute="", description="")})
+public class GrokParser extends AbstractProcessor {
--- End diff --

The naming convention that we try to stick with for Processors is 
. While this may be counter-intuitive for a Java Developer, it 
results in making the flow much more readable for users. So we should consider 
ParseLog or GrokLog.


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572098#comment-15572098
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83229352
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGrokParser.java
 ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+/**
+ * Created by snamsi on 05/10/16.
+ */
+public class TestGrokParser {
+
+private TestRunner testRunner;
+final static Path GROK_LOG_INPUT = 
Paths.get("src/test/resources/TestGrokParser/apache.log");
+final static Path GROK_TEXT_INPUT = 
Paths.get("src/test/resources/TestGrokParser/simple_text.log");
+
+
+@Before
+public void init() {
+testRunner = TestRunners.newTestRunner(GrokParser.class);
+}
+
+@Test
+public void testGrokParserWithMatchedContent() throws IOException {
+
+
+testRunner.setProperty(GrokParser.GROK_EXPRESSION, 
"%{COMMONAPACHELOG}");
+testRunner.setProperty(GrokParser.GROK_PATTERN_FILE, 
"src/test/resources/TestGrokParser/patterns");
+testRunner.enqueue(GROK_LOG_INPUT);
+testRunner.run();
+testRunner.assertAllFlowFilesTransferred(GrokParser.REL_MATCH);
+final MockFlowFile matched = 
testRunner.getFlowFilesForRelationship(GrokParser.REL_MATCH).get(0);
+
+matched.assertAttributeEquals("verb","GET");
+matched.assertAttributeEquals("response","401");
+matched.assertAttributeEquals("bytes","12846");
+matched.assertAttributeEquals("clientip","64.242.88.10");
+matched.assertAttributeEquals("auth","-");
+matched.assertAttributeEquals("timestamp","07/Mar/2004:16:05:49 
-0800");
+
matched.assertAttributeEquals("request","/twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables");
+matched.assertAttributeEquals("httpversion","1.1");
+
+}
+
+@Test
+public void testGrokParserWithUnMatchedContent() throws IOException {
+
+
+testRunner.setProperty(GrokParser.GROK_EXPRESSION, "%{ADDRESS}");
+testRunner.setProperty(GrokParser.GROK_PATTERN_FILE, 
"src/test/resources/TestGrokParser/patterns");
+testRunner.enqueue(GROK_TEXT_INPUT);
+testRunner.run();
+testRunner.assertAllFlowFilesTransferred(GrokParser.REL_NO_MATCH);
+final MockFlowFile notMatched = 
testRunner.getFlowFilesForRelationship(GrokParser.REL_NO_MATCH).get(0);
+notMatched.assertContentEquals(GROK_TEXT_INPUT);
+
+}
+
+@Test(expected = java.lang.AssertionError.class)
+public void testGrokParserWithNotFoundPatternFile() throws IOException 
{
+
+testRunner.setProperty(GrokParser.GROK_EXPRESSION, 
"%{COMMONAPACHELOG}");
+testRunner.setProperty(GrokParser.GROK_PATTERN_FILE, 
"src/test/resources/TestGrokParser/toto_file");
+testRunner.enqueue(GROK_LOG_INPUT);
+testRunner.run();
+
+}
+
+
+@Test(expected = java.lang.AssertionError.class)
--- End diff --

Rather than expected an AssertionError, we should avoid calling 
testRunner.run() and instead just use testRunner.assertNotValid()


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> 

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572087#comment-15572087
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83226436
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GrokParser.java
 ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.FlowFile;
+
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.StreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+
+
+@Tags({"Grok Processor"})
+@CapabilityDescription("Use Grok expression ,a la logstash, to parse 
data.")
--- End diff --

We should probably expand on this a bit more. Many users will not know what 
Grok is.


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572103#comment-15572103
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83229592
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestGrokParser/patterns
 ---
@@ -0,0 +1,108 @@
+# Forked from 
https://github.com/elasticsearch/logstash/tree/v1.4.0/patterns
--- End diff --

We have to ensure that we have proper licensing for these test files.


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572097#comment-15572097
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1108#discussion_r83229335
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGrokParser.java
 ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+/**
+ * Created by snamsi on 05/10/16.
+ */
+public class TestGrokParser {
+
+private TestRunner testRunner;
+final static Path GROK_LOG_INPUT = 
Paths.get("src/test/resources/TestGrokParser/apache.log");
+final static Path GROK_TEXT_INPUT = 
Paths.get("src/test/resources/TestGrokParser/simple_text.log");
+
+
+@Before
+public void init() {
+testRunner = TestRunners.newTestRunner(GrokParser.class);
+}
+
+@Test
+public void testGrokParserWithMatchedContent() throws IOException {
+
+
+testRunner.setProperty(GrokParser.GROK_EXPRESSION, 
"%{COMMONAPACHELOG}");
+testRunner.setProperty(GrokParser.GROK_PATTERN_FILE, 
"src/test/resources/TestGrokParser/patterns");
+testRunner.enqueue(GROK_LOG_INPUT);
+testRunner.run();
+testRunner.assertAllFlowFilesTransferred(GrokParser.REL_MATCH);
+final MockFlowFile matched = 
testRunner.getFlowFilesForRelationship(GrokParser.REL_MATCH).get(0);
+
+matched.assertAttributeEquals("verb","GET");
+matched.assertAttributeEquals("response","401");
+matched.assertAttributeEquals("bytes","12846");
+matched.assertAttributeEquals("clientip","64.242.88.10");
+matched.assertAttributeEquals("auth","-");
+matched.assertAttributeEquals("timestamp","07/Mar/2004:16:05:49 
-0800");
+
matched.assertAttributeEquals("request","/twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables");
+matched.assertAttributeEquals("httpversion","1.1");
+
+}
+
+@Test
+public void testGrokParserWithUnMatchedContent() throws IOException {
+
+
+testRunner.setProperty(GrokParser.GROK_EXPRESSION, "%{ADDRESS}");
+testRunner.setProperty(GrokParser.GROK_PATTERN_FILE, 
"src/test/resources/TestGrokParser/patterns");
+testRunner.enqueue(GROK_TEXT_INPUT);
+testRunner.run();
+testRunner.assertAllFlowFilesTransferred(GrokParser.REL_NO_MATCH);
+final MockFlowFile notMatched = 
testRunner.getFlowFilesForRelationship(GrokParser.REL_NO_MATCH).get(0);
+notMatched.assertContentEquals(GROK_TEXT_INPUT);
+
+}
+
+@Test(expected = java.lang.AssertionError.class)
--- End diff --

Rather than expected an AssertionError, we should avoid calling 
testRunner.run() and instead just use testRunner.assertNotValid()


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572044#comment-15572044
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1108
  
@selim-namsi Thanks for contributing this! I have actually been very 
interested in using NiFi to do some log parsing but hadn't really dug in very 
much to understand the best way to go about it. This looks like it could be 
very powerful!

Before we get this merged into the codebase, though, it looks like there is 
some work that needs to be done to the PR. The concern stems, I think, from you 
not yet being overly familiar with the API, as there are empty 
@ReadsAttributes, @WritesAttributes annotations, etc. But the great news is 
that the NiFi community tends to be very inclusive and will help to get 
everything in great shape!

One thing that I did notice is that you updated the Licensing information, 
which is one of the most commonly overlooked issues. So very glad that's there. 
I'll leave some inline feedback on things that I notice, but very much looking 
forward to this getting in!


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563873#comment-15563873
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user selim-namsi commented on the issue:

https://github.com/apache/nifi/pull/1108
  
@trixpan I figured out how to rebase the pull request :)

Cheers


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558860#comment-15558860
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

GitHub user selim-namsi reopened a pull request:

https://github.com/apache/nifi/pull/1108

NIFI-2565: add Grok parser

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/selim-namsi/nifi nifi-2565

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1108.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1108


commit 447c65ec272fd72b8b55a015f36449e387400fe6
Author: Selim Namsi 
Date:   2016-10-05T16:54:37Z

nifi-2565: add Grok parser

commit 1dffac47057c85c7aba2e0b2a8543eafc88e96be
Author: Selim Namsi 
Date:   2016-10-05T17:17:49Z

nifi-2656: Update LICENSE after adding Grok Parser

commit c764f83506165c99e77c707e358cb327e9af16b8
Author: Scott Aslan 
Date:   2016-10-05T15:46:48Z

[NIFI-1904] If open, close hamburger menu on window resize. This closes 
#1106

commit e46fea920af11d2527f4eb4da3a191e309d12618
Author: Scott Aslan 
Date:   2016-10-04T18:13:17Z

[NIFI-1794] Update .dialog-content to wrap text. This closes #1094

commit da33e2859ce45321d28901e5820c38a37dcfc709
Author: mans2singh 
Date:   2016-07-24T23:18:55Z

NIFI-2398 - GetIgnite processor

This closes #721.

commit 6ad633d17422f2110645553cb03ae3b364926eee
Author: Scott Aslan 
Date:   2016-10-05T20:00:34Z

[NIFI-2838] update width of rule name and save message. This closes #1089

commit 6f1af31ff28f60d0eddbee5dafe909bc66cc9c71
Author: Joe N 
Date:   2016-10-03T22:16:14Z

NIFI-2852 base64 expression language functions

Signed-off-by: jpercivall 

commit 3c8545a90266b8f82a6c541c9a68daad107b0f23
Author: Pierre Villard 
Date:   2016-09-29T20:47:04Z

NIFI-1912 - PutEmail fixed format when attachment

Correction as suggested by users in JIRA.
+ adding a unit test to check attachments.

This closes: #1081

Signed-off-by: Andre F de Miranda 

commit 3c673972e035f1168e509128150444da78af5292
Author: Matt Gilman 
Date:   2016-10-06T13:29:59Z

NIFI-2816:
- Fixing compilation error resulting from the initial NIFI-2816 commit.

commit 8bd85e20853c44bbb33bcad2795bfb4ac8819e1a
Author: Scott Aslan 
Date:   2016-10-05T20:17:32Z

[NIFI-1792] Clear the selected rule id when deleting the last rule in the 
list. Add scrollable styles when appropriate. Close popups when appropriate. 
This PR also adjusts the position of the table cell nfel and long text editors. 
This closes #1099.

commit 92cca96d49042f9898f93b3a2d2210b924708e52
Author: Mark Payne 
Date:   

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558858#comment-15558858
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user selim-namsi commented on the issue:

https://github.com/apache/nifi/pull/1108
  
@trixpan  could you please tell me how to rebase a PR ? I rebased my 
[branch](https://github.com/selim-namsi/nifi/commits/nifi-2565) but I didn't 
find how to rebase the PR.
sorry for the inconvenience , this is my first PR.

Cheers


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558859#comment-15558859
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user selim-namsi closed the pull request at:

https://github.com/apache/nifi/pull/1108


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558785#comment-15558785
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1108
  
Please rebase this PR so it only include your changes. 

Cheers


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553486#comment-15553486
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1108
  
By the way, most of these modifications are already available here: 

https://github.com/trixpan/nifi/commits/NIFI-2565


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553478#comment-15553478
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1108
  
You can certainly include a file with the default patterns, however you 
should not hardcode them. By hardcoding them you prevent the user from 
optimising for speed by removing unused patterns from the pattern files (I 
realise they can remove from the default packaged patterns but that means they 
would be changing packaged files after an install, something you should always 
avoid) 


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553464#comment-15553464
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user selim-namsi commented on the issue:

https://github.com/apache/nifi/pull/1108
  
@trixpan Thank you for all this useful feedback, I'll start working on 
these modifications.
For the patterns, I hard coded the patterns because I was thinking about 
adding by default some useful patterns and also let the user add his custom 
pattern. What do you think about it ?

Thanks


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550144#comment-15550144
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1108
  
@selim-namsi thanks for putting together. I was coding this processor but 
happy to review it.

Few comments:

1. Your code is failing -Pcontrib-check. Can you please fix this?

2. Ideally I believe this processor should allow use to chose between 
content replacement (replace the original log line with the json 
representation) and adding attributes (what you already did). This should give 
an idea of what I mean: https://github.com/apache/nifi/pull/785

3. Please don't hard-code the patterns, instead, let the user configure the 
pattern files

4. The java-grok version has some bugs, you may want to upgrade it (this is 
the reason I haven't submitted the code previously... :smiley: )

Thank you again, looking forward your modifications


> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549393#comment-15549393
 ] 

ASF GitHub Bot commented on NIFI-2565:
--

GitHub user selim-namsi opened a pull request:

https://github.com/apache/nifi/pull/1108

NIFI-2565: add Grok parser

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/selim-namsi/nifi nifi-2565

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1108.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1108


commit 447c65ec272fd72b8b55a015f36449e387400fe6
Author: Selim Namsi 
Date:   2016-10-05T16:54:37Z

nifi-2565: add Grok parser

commit 1dffac47057c85c7aba2e0b2a8543eafc88e96be
Author: Selim Namsi 
Date:   2016-10-05T17:17:49Z

nifi-2656: Update LICENSE after adding Grok Parser




> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2565) NiFi processor to parse logs using Grok patterns

[Andre (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548148#comment-15548148
 ] 

Andre commented on NIFI-2565:
-

Found a processor but developer did not return contact regarding a possible PR.

Coding one completely from from scratch.

> NiFi processor to parse logs using Grok patterns
> 
>
> Key: NIFI-2565
> URL: https://issues.apache.org/jira/browse/NIFI-2565
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
> Fix For: 1.1.0
>
>
> Following up on Ryan Ward to create a Grok capable parser
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201606.mbox/%3CCADD=rnPa8nHkJbeM280=PTQ=wurtwhstm5u+7btoo9pcym2...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)