[
https://issues.apache.org/jira/browse/NIFI-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562689#comment-15562689
]
ASF GitHub Bot commented on NIFI-2341:
--------------------------------------
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/785#discussion_r82634033
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestParseCEF.java
---
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.TimeZone;
+
+
+public class TestParseCEF {
+ private SimpleDateFormat sdf = new
SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ");
+
+ private final static String sample1 =
"CEF:0|TestVendor|TestProduct|TestVersion|TestEventClassID|TestName|Low|" +
+ // TimeStamp, String and Long
+ "rt=Feb 09 2015 00:27:43 UTC cn3Label=Test Long
cn3=9223372036854775807 " +
+ // FloatPoint and MacAddress
+ "cfp1=1.234 cfp1Label=Test FP Number smac=00:00:0c:07:ac:00 " +
+ // IPv6 and String
+ "c6a3=2001:cdba::3257:9652 c6a3Label=Test IPv6 " +
+ // IPv4
+ "destinationTranslatedAddress=123.123.123.123 " +
+ // Date without TZ
+ "deviceCustomDate1=Feb 06 2015 13:27:43 " +
+ // Integer and IP Address (from v4)
+ "dpt=1234 agt=123.123.0.124 dlat=40.366633";
+
+ @Test
+ public void testInvalidMessage() {
+ final TestRunner runner = TestRunners.newTestRunner(new
ParseCEF());
+ runner.enqueue("test test test chocolate\n".getBytes());
+ runner.run();
+
+ runner.assertAllFlowFilesTransferred(ParseCEF.REL_FAILURE, 1);
+ }
+
+ @Test
+ public void testSuccessfulParseToAttributes() throws IOException {
--- End diff --
I could get your example CEF file to parse correctly, but I had trouble
with the following two inputs I got from the internet:
`127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET
/apache_pb.gif HTTP/1.0" 200 2326`
(https://httpd.apache.org/docs/trunk/logs.html#common)
`CEF:0|ArcSight|ArcSight|6.0.3.6664.0|agent:030|Agent [test] type
[testalertng] started|Low| eventId=1 mrt=1396328238973
categorySignificance=/Normal categoryBehavior=/Execute/Start
categoryDeviceGroup=/Application catdt=Security Mangement
categoryOutcome=/Success categoryObject=/Host/Application/Service
art=1396328241038 cat=/Agent/Started deviceSeverity=Warning rt=1396328238937
fileType=Agent cs2=<Resource ID\="3DxKlG0UBABCAA0cXXAZIwA\=\="/>
c6a4=fe80:0:0:0:495d:cc3c:db1a:de71 cs2Label=Configuration Resource
c6a4Label=Agent IPv6 Address ahost=SKEELES10 agt=888.99.100.1 agentZoneURI=/All
Zones/ArcSight System/Private Address Space Zones/RFC1918:
888.99.0.0-888.200.255.255 av=6.0.3.6664.0 atz=Australia/Sydney
aid=3DxKlG0UBABCAA0cXXAZIwA\=\= at=testalertng dvchost=SKEELES10
dvc=888.99.100.1 deviceZoneURI=/All Zones/ArcSight System/Private Address Space
Zones/RFC1918: 888.99.0.0-888.200.255.255 dtz=Australia/Sydney _cefVer=0.1`
(https://my.vertica.com/docs/7.1.x/HTML/Content/Authoring/FlexTables/LoadCEFData.htm)
In both cases it said the file could not be parsed as it was not in CEF
format. I presumed the first one is missing the CEF:0 header, and maybe the
second one has weird characters? I removed all newlines so that second example
is all on one line.
> Create a processor to parse logs formated using CEF
> ---------------------------------------------------
>
> Key: NIFI-2341
> URL: https://issues.apache.org/jira/browse/NIFI-2341
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Andre
> Assignee: Andre
> Fix For: 1.1.0
>
>
> As NiFi continue to increase its abilities to complement SIEM, Splunk and ELK
> deployments, a number of users will be looking to parse CEF formatted
> logs[1][2].
> CEF is a format specified by Arcsight (now part of HPE) and is described in
> detail in here:
> https://www.protect724.hpe.com/docs/DOC-1072
> [1]
> http://apache-nifi.1125220.n5.nabble.com/Suggestion-of-processors-td9795.html
> [2]
> https://community.hortonworks.com/questions/43185/which-processor-is-used-to-parse-cef-format-logs.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
