[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15596183#comment-15596183
 ] 

ASF subversion and git services commented on NIFI-2341:
---

Commit b864d49f66692dabeb805965a08e83e14d27fd58 in nifi's branch 
refs/heads/appveyor-improvement from Andre F de Miranda
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b864d49 ]

NIFI-2341 - Introduce ParseCEF processor

This closes #785


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593041#comment-15593041
 ] 

ASF subversion and git services commented on NIFI-2341:
---

Commit b864d49f66692dabeb805965a08e83e14d27fd58 in nifi's branch 
refs/heads/master from Andre F de Miranda
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b864d49 ]

NIFI-2341 - Introduce ParseCEF processor

This closes #785


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593042#comment-15593042
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/785


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593039#comment-15593039
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/785
  
+1 LGTM, thanks much! Merging to master.


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577375#comment-15577375
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/785
  
@mattyb149 hopefully all addressed


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577291#comment-15577291
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r83525194
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseCEF.java
 ---
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonSerializer;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fluenda.parcefone.event.CEFHandlingException;
+import com.fluenda.parcefone.event.CommonEvent;
+import com.fluenda.parcefone.parser.CEFParser;
+
+import com.martiansoftware.macnificent.MacAddress;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.time.ZoneId;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TimeZone;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "cef", "attributes", "system", "event", "message"})
+@CapabilityDescription("Parses the contents of a CEF formatted message and 
adds attributes to the FlowFile for " +
+"headers and extensions of the parts of the CEF message.\n" +
+"Note: This Processor expects CEF messages WITHOUT the syslog 
headers (i.e. starting at \"CEF:0\"")
+@WritesAttributes({@WritesAttribute(attribute = "cef.header.version", 
description = "The version of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceVendor", description = 
"The Device Vendor of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceProduct", description = 
"The Device Product of the CEF message."),
+@WritesAttribute(attribute = 

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563484#comment-15563484
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r82681523
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseCEF.java
 ---
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonSerializer;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fluenda.parcefone.event.CEFHandlingException;
+import com.fluenda.parcefone.event.CommonEvent;
+import com.fluenda.parcefone.parser.CEFParser;
+
+import com.martiansoftware.macnificent.MacAddress;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.time.ZoneId;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TimeZone;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "cef", "attributes", "system", "event", "message"})
+@CapabilityDescription("Parses the contents of a CEF formatted message and 
adds attributes to the FlowFile for " +
+"headers and extensions of the parts of the CEF message.\n" +
+"Note: This Processor expects CEF messages WITHOUT the syslog 
headers (i.e. starting at \"CEF:0\"")
+@WritesAttributes({@WritesAttribute(attribute = "cef.header.version", 
description = "The version of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceVendor", description = 
"The Device Vendor of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceProduct", description = 
"The Device Product of the CEF message."),
+@WritesAttribute(attribute = 

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562688#comment-15562688
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r82633365
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseCEF.java
 ---
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonSerializer;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fluenda.parcefone.event.CEFHandlingException;
+import com.fluenda.parcefone.event.CommonEvent;
+import com.fluenda.parcefone.parser.CEFParser;
+
+import com.martiansoftware.macnificent.MacAddress;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.time.ZoneId;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TimeZone;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "cef", "attributes", "system", "event", "message"})
+@CapabilityDescription("Parses the contents of a CEF formatted message and 
adds attributes to the FlowFile for " +
+"headers and extensions of the parts of the CEF message.\n" +
+"Note: This Processor expects CEF messages WITHOUT the syslog 
headers (i.e. starting at \"CEF:0\"")
+@WritesAttributes({@WritesAttribute(attribute = "cef.header.version", 
description = "The version of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceVendor", description = 
"The Device Vendor of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceProduct", description = 
"The Device Product of the CEF message."),
+@WritesAttribute(attribute = 

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537363#comment-15537363
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/785
  
Sounds good thanks, will take a look soon


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537361#comment-15537361
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/785
  
@mattyb149 all feedback addressed.

this should be ready for review


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529880#comment-15529880
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r80934511
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml ---
@@ -254,6 +254,17 @@ language governing permissions and limitations under 
the License. -->
 org.everit.json.schema
 1.4.0
 
+
+com.fluenda
+ParCEFone
--- End diff --

both done.


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526464#comment-15526464
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r80719887
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml ---
@@ -254,6 +254,17 @@ language governing permissions and limitations under 
the License. -->
 org.everit.json.schema
 1.4.0
 
+
+com.fluenda
+ParCEFone
--- End diff --

L are areas we must be picky so I wil be happy to address the NOTICE. I 
promise one day I will get this thing right. :smiley: 

I will check if I can upgrade to 3.0.0 so to reduce the amount of 
"paperwork".



> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526362#comment-15526362
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r80709952
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml ---
@@ -254,6 +254,17 @@ language governing permissions and limitations under 
the License. -->
 org.everit.json.schema
 1.4.0
 
+
+com.fluenda
+ParCEFone
--- End diff --

Sorry to be overly picky. This library uses javax.el-api which is licensed 
under CDDL 1.1, and its NOTICE has been copied to the NiFi overall NOTICE and 
the assembly's NOTICE. However it is not in the standard-nar's notice 
(nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/src/main/resources/META-INF/NOTICE).
 This should have already been included here since the Jolt UI uses it, but 
since it has not been included, do you mind adding it? Just need to add the 
following line to the CDDL 1.1 section of the aforementioned file:

(CDDL 1.1) (GPL2 w/ CPE) Expression Language 2.2.4 API  
(javax.el:javax.el-api:jar:2.2.4 - http://uel-spec.java.net)

Note that the Jolt UI uses el-api version 3.0.0 but your library is using 
2.2.4. This will cause two different versions (in two different areas once 
unpacked, so no eviction or other issues), but as a NAR we would want either 
two entries (one for each version), or perhaps consider upgrading your library 
to use 3.0.0.




> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526287#comment-15526287
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r80703541
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestParseCEF.java
 ---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.TimeZone;
+
+
+public class TestParseCEF {
+private SimpleDateFormat sdf = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSZ");
+
+private final static String sample1 = 
"CEF:0|TestVendor|TestProduct|TestVersion|TestEventClassID|TestName|Low|" +
+// TimeStamp, String and Long
+"rt=Feb 09 2015 00:27:43 UTC cn3Label=Test Long 
cn3=9223372036854775807 " +
+// FloatPoint and MacAddress
+"cfp1=1.234 cfp1Label=Test FP Number smac=00:00:0c:07:ac:00 " +
+// IPv6 and String
+"c6a3=2001:cdba::3257:9652 c6a3Label=Test IPv6 " +
+// IPv4
+"destinationTranslatedAddress=123.123.123.123 " +
+// Date without TZ
+"deviceCustomDate1=Feb 06 2015 13:27:43 " +
+// Integer  and IP Address (from v4)
+"dpt=1234 agt=123.123.0.124 dlat=40.366633";
+
+@Test
+public void testInvalidMessage() {
--- End diff --

If full or no validation is the behavior of the library then no need for 
extra unit tests here, I will run it on a real system and merge if all looks 
well :)


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525724#comment-15525724
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r80663321
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseCEF.java
 ---
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonSerializer;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fluenda.parcefone.event.CEFHandlingException;
+import com.fluenda.parcefone.event.CommonEvent;
+import com.fluenda.parcefone.parser.CEFParser;
+
+import com.martiansoftware.macnificent.MacAddress;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.time.ZoneId;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TimeZone;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "cef", "attributes", "system", "event", "message"})
+@CapabilityDescription("Parses the contents of a CEF formatted message and 
adds attributes to the FlowFile for " +
+"headers and extensions of the parts of the CEF message.\n" +
+"Note: This Processor expects CEF messages WITHOUT the syslog 
headers (i.e. starting at \"CEF:0\"")
+@WritesAttributes({@WritesAttribute(attribute = "cef.header.version", 
description = "The version of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceVendor", description = 
"The Device Vendor of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceProduct", description = 
"The Device Product of the CEF message."),
+@WritesAttribute(attribute = 

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525720#comment-15525720
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r80663050
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestParseCEF.java
 ---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.TimeZone;
+
+
+public class TestParseCEF {
+private SimpleDateFormat sdf = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSZ");
+
+private final static String sample1 = 
"CEF:0|TestVendor|TestProduct|TestVersion|TestEventClassID|TestName|Low|" +
+// TimeStamp, String and Long
+"rt=Feb 09 2015 00:27:43 UTC cn3Label=Test Long 
cn3=9223372036854775807 " +
+// FloatPoint and MacAddress
+"cfp1=1.234 cfp1Label=Test FP Number smac=00:00:0c:07:ac:00 " +
+// IPv6 and String
+"c6a3=2001:cdba::3257:9652 c6a3Label=Test IPv6 " +
+// IPv4
+"destinationTranslatedAddress=123.123.123.123 " +
+// Date without TZ
+"deviceCustomDate1=Feb 06 2015 13:27:43 " +
+// Integer  and IP Address (from v4)
+"dpt=1234 agt=123.123.0.124 dlat=40.366633";
+
+@Test
+public void testInvalidMessage() {
--- End diff --

I can certainly think about invalid messages but generally speaking this is 
ParCEFone's land. In the library did I either do full validation or no 
validation at all (we use full validation in ParseCEF).

Happy to port the jUnits across to NIFI.

One thing worth of notice is that the positive sample above contains all 
the data types that ParCEFone can process (i.e. String, float, long, 
MacAddress, IPAddr (v4 and v6) and timestamps).


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523777#comment-15523777
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r80535518
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseCEF.java
 ---
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonSerializer;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fluenda.parcefone.event.CEFHandlingException;
+import com.fluenda.parcefone.event.CommonEvent;
+import com.fluenda.parcefone.parser.CEFParser;
+
+import com.martiansoftware.macnificent.MacAddress;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.BufferedOutputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.time.ZoneId;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TimeZone;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "cef", "attributes", "system", "event", "message"})
+@CapabilityDescription("Parses the contents of a CEF formatted message and 
adds attributes to the FlowFile for " +
+"headers and extensions of the parts of the CEF message.\n" +
+"Note: This Processor expects CEF messages WITHOUT the syslog 
headers (i.e. starting at \"CEF:0\"")
+@WritesAttributes({@WritesAttribute(attribute = "cef.header.version", 
description = "The version of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceVendor", description = 
"The Device Vendor of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceProduct", description = 
"The Device Product of the CEF message."),
+@WritesAttribute(attribute = 

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523776#comment-15523776
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r80535990
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestParseCEF.java
 ---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.TimeZone;
+
+
+public class TestParseCEF {
+private SimpleDateFormat sdf = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSZ");
+
+private final static String sample1 = 
"CEF:0|TestVendor|TestProduct|TestVersion|TestEventClassID|TestName|Low|" +
+// TimeStamp, String and Long
+"rt=Feb 09 2015 00:27:43 UTC cn3Label=Test Long 
cn3=9223372036854775807 " +
+// FloatPoint and MacAddress
+"cfp1=1.234 cfp1Label=Test FP Number smac=00:00:0c:07:ac:00 " +
+// IPv6 and String
+"c6a3=2001:cdba::3257:9652 c6a3Label=Test IPv6 " +
+// IPv4
+"destinationTranslatedAddress=123.123.123.123 " +
+// Date without TZ
+"deviceCustomDate1=Feb 06 2015 13:27:43 " +
+// Integer  and IP Address (from v4)
+"dpt=1234 agt=123.123.0.124 dlat=40.366633";
+
+@Test
+public void testInvalidMessage() {
--- End diff --

Are there any other inputs or properties that could cause a different 
error? If so additional tests would be nice (for illustration and regression 
purposes).


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523775#comment-15523775
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r80535718
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutSQL.java
 ---
@@ -819,7 +819,7 @@ private void setParameter(final PreparedStatement stmt, 
final String attrName, f
 bValue = 
DatatypeConverter.parseBase64Binary(parameterValue);
 break;
 default:
-throw new ParseException("Unable to parse 
binary data using the formatter `" + valueFormat + "`.",0);
+throw new ParseException("Unable to parse 
binary data using the simpleDateFormat `" + valueFormat + "`.",0);
--- End diff --

This change looks odd (no other reference to such things), perhaps a rebase 
is needed? Or maybe a search-and-replace ran rampant and this file should be 
removed from the PR?


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516205#comment-15516205
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/785
  
@mattyb149 PR rebased and in theory should be ready for review 


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459979#comment-15459979
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/785
  
Rofl. Well at least I  got 1 out 3 right. As you can see we security folks 
are a bit obsessed with threats and risks :-) 


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459932#comment-15459932
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/785
  
Will do. Although I'm sure you meant "thread concurrency", I always keep an 
eye out for threat concurrency ;) I've just started a long holiday weekend so I 
may not be able to take a look until mid-next-week. Thanks for the updates!


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459917#comment-15459917
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/785
  
@mattyb149 

feedback addressed. 

My comment is to play particular attention on threat concurrency around 
Jackson's ObjectMapper  (jackson say it is threat safe but thread safety is not 
my forté) and the fiddling with timezones (that has played tricks in the past).

I will continue testing but overall I suspect it should be ok to review. 


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15445965#comment-15445965
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r76612149
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseCEF.java
 ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fluenda.parcefone.event.CEFHandlingException;
+import com.fluenda.parcefone.event.CommonEvent;
+import com.fluenda.parcefone.parser.CEFParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "cef", "attributes", "system", "event", "message"})
+@CapabilityDescription("Parses the contents of a CEF formatted message and 
adds attributes to the FlowFile for " +
+"headers and extensions of the parts of the CEF message.\n" +
+"Note: This Processor expects CEF messages WITHOUT the syslog 
headers (i.e. starting at \"CEF:0\"")
+@WritesAttributes({@WritesAttribute(attribute = "cef.header.version", 
description = "The version of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceVendor", description = 
"The Device Vendor of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceProduct", description = 
"The deviceProduct of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceVersion", description = 
"The deviceVersion of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceEventClassId", 
description = "The deviceEventClassId of the CEF message."),
+@WritesAttribute(attribute = "cef.header.name", description = "The 
name of the CEF message."),
+@WritesAttribute(attribute = "cef.header.severity", description = "The 
severity of the CEF message."),
+@WritesAttribute(attribute = "cef.extension.*", description = "The key 
and value generated by the parsing of the message.")})
+@SeeAlso({ParseSyslog.class})
+
+public class ParseCEF extends AbstractProcessor {
+
+
+
+static final Relationship REL_FAILURE = new Relationship.Builder()
+.name("failure")
+.description("Any FlowFile that could not be parsed as a CEF 
message will be transferred to this Relationship without any attributes being 
added")
+.build();
+static final Relationship REL_SUCCESS = new 

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15445948#comment-15445948
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r76611496
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseCEF.java
 ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fluenda.parcefone.event.CEFHandlingException;
+import com.fluenda.parcefone.event.CommonEvent;
+import com.fluenda.parcefone.parser.CEFParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "cef", "attributes", "system", "event", "message"})
+@CapabilityDescription("Parses the contents of a CEF formatted message and 
adds attributes to the FlowFile for " +
+"headers and extensions of the parts of the CEF message.\n" +
+"Note: This Processor expects CEF messages WITHOUT the syslog 
headers (i.e. starting at \"CEF:0\"")
+@WritesAttributes({@WritesAttribute(attribute = "cef.header.version", 
description = "The version of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceVendor", description = 
"The Device Vendor of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceProduct", description = 
"The deviceProduct of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceVersion", description = 
"The deviceVersion of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceEventClassId", 
description = "The deviceEventClassId of the CEF message."),
+@WritesAttribute(attribute = "cef.header.name", description = "The 
name of the CEF message."),
+@WritesAttribute(attribute = "cef.header.severity", description = "The 
severity of the CEF message."),
+@WritesAttribute(attribute = "cef.extension.*", description = "The key 
and value generated by the parsing of the message.")})
--- End diff --

How big are CEF messages? Is is prudent to put these as attributes (which 
are in-memory and not copy-on-write like content IIRC)? Maybe a choice could be 
offered for the target (as is done in other processors), such as 
"flowfile-attribute" and "flowfile-content". If attributes were selected they 
could be populated like they are now, if content was selected perhaps the 
processor writes a flat JSON object. That way, if the 

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15445937#comment-15445937
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/785#discussion_r76610727
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseCEF.java
 ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fluenda.parcefone.event.CEFHandlingException;
+import com.fluenda.parcefone.event.CommonEvent;
+import com.fluenda.parcefone.parser.CEFParser;
+
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "cef", "attributes", "system", "event", "message"})
+@CapabilityDescription("Parses the contents of a CEF formatted message and 
adds attributes to the FlowFile for " +
+"headers and extensions of the parts of the CEF message.\n" +
+"Note: This Processor expects CEF messages WITHOUT the syslog 
headers (i.e. starting at \"CEF:0\"")
+@WritesAttributes({@WritesAttribute(attribute = "cef.header.version", 
description = "The version of the CEF message."),
+@WritesAttribute(attribute = "cef.header.deviceVendor", description = 
"The Device Vendor of the CEF message."),
--- End diff --

Cosmetic note, the description of the deviceVendor attribute uses proper 
nouns but the rest of the device* properties do not.


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15441842#comment-15441842
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/785
  
Ready for review


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15441615#comment-15441615
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/785
  
figured out what it is. I needed to exclude slf4j-log4j12 from parCEFone 
dependency.

In the process found a few bugs. Fixing. :-)


> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15441557#comment-15441557
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/785
  
@mattyb149 - For some reason I don't truly grasp TestGetJMSQueue start to 
fail once this processor is introduced to standard-processors

Other than that code looks to be working as expected




> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407771#comment-15407771
 ] 

ASF GitHub Bot commented on NIFI-2341:
--

GitHub user trixpan opened a pull request:

https://github.com/apache/nifi/pull/785

NIFI-2341 - Introduce ParseCEF processor

* Implements a processor to parse HPE's highly popular Common Event Format

* Please note this code relies on a Parser written outside the NiFi source 
tree. This was done with the intent of allowing code reuse, given that many 
Open Source projects tend to tackle the CEF format in a myriad of ways and some 
are still trying to implement a solid parser (e.g. METRON-157). The Parser is 
functional but feedback over its implementation is welcome as well.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/trixpan/nifi NIFI-2341

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/785.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #785


commit be2c1e83f7f486b1b6a1e148bc828acf3865a8de
Author: Andre F de Miranda 
Date:   2016-08-04T13:30:37Z

NIFI-2341 - Introduce ParseCEF processor




> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2341) Create a processor to parse logs formated using CEF

[Andre (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403359#comment-15403359
 ] 

Andre commented on NIFI-2341:
-

Created a library to parse and validate CEF logs

https://github.com/fluenda/ParCEFone

Now will use it to create the processor

> Create a processor to parse logs formated using CEF
> ---
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre
>Assignee: Andre
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK 
> deployments, a number of users will be looking to parse CEF formatted 
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in 
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1] 
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2] 
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)