[
https://issues.apache.org/jira/browse/METRON-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15876097#comment-15876097
]
ASF GitHub Bot commented on METRON-157:
---------------------------------------
GitHub user simonellistonball reopened a pull request:
https://github.com/apache/incubator-metron/pull/451
METRON-157: Added CEF Parser
There is some discussion of using an external library on the jira ticket
for this issue. The library in question is excellent, and covers the spec well,
but applies types which don't get along with Metron (IP address, MAC address).
This has a simplified parser focussing on speed and sticking to basic types.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/simonellistonball/incubator-metron METRON-157
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/451.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #451
----
commit 8074a74d0ea6370fef4fb8b4e74dded63eae5c4d
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-11T16:28:39Z
Added CEF Parser
commit 6f50b1756046d63234e74230118d0a12b0abdce4
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-11T16:41:53Z
Changed guava version for parsers module to fix tests
commit ce082dff2d5a2878df34c8d279a071fa5e05dc78
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-11T17:07:09Z
Added License to sources and fixed guava import
commit f783f99209d740bd9dbe0b79dbd5b0895d2d8588
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-11T23:42:01Z
Added License
commit 792a19ae30e85c5eca9fd86f23bd641647cf4d08
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-11T23:42:59Z
Additional CEF formats per specfication
commit c6174b7a1d5af8821d11291d44c351cc2c9a95f4
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-12T10:52:13Z
Replaced domain names with example.com
commit c71231fea9806759b5bb912956668920793a9b40
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-12T10:56:59Z
Fixed priority of rt, syslog and current time
commit f9e813f41e17503de7345becc6898dbc39082c09
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-12T11:53:32Z
Added tests for timestamp priority and syslog parsing
commit cc53d02070b548c266a5d402ce322d342c591d03
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-14T00:59:58Z
Added Test for rt field being unix epoch
commit 110a0f0f412d98520b13b008e266c14e44509845
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-14T01:09:10Z
Added support for parsing timestamp based dates to the DateUtils class
commit d2eb8296a57c77380756a27b55d6cd289440611d
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-14T01:29:10Z
Added test for missing year in CEF date formats
commit c8ab7855d4e9b24e53f5397199b8ec963e4c05b2
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-18T09:44:06Z
Fixed date parsing logic to apply 4 day in future rule and missing year rule
commit 7a3036c7f4a4ced2333780a372f1386403292126
Author: Simon Elliston Ball <[email protected]>
Date: 2017-02-18T10:06:35Z
Added additional CEF sample test from NiFi parser to test JSON in CEF
support
----
> Create CEF Parser
> -----------------
>
> Key: METRON-157
> URL: https://issues.apache.org/jira/browse/METRON-157
> Project: Metron
> Issue Type: New Feature
> Reporter: Domenic Puzio
> Priority: Minor
> Labels: platform
>
> Create a parser for CEF (Common Event Format). CEF is a very common
> formatting for security data sources; it is used by FireEye, Adallom, Imperva
> WAF, CyberArk, and others. The parser should be flexible enough to work for
> any of these data sources. CEF uses shorthand field names, so field names
> should be changed to human-readable and Metron-friendly equivalents. CEF
> custom labels (cs1Label, flexString1Label, etc.) should be converted
> appropriately.
> Below are sample messages and their expected parsed output.
> Adallom CEF
> 2016-04-01T09:29:11.356-0400
> CEF:0|Adallom|Adallom|1.0|56fe779ee4b0459f4e9a484a|ALERT_CABINET_EVENT_MATCH_AUDIT|0|msg=Activity
> policy 'User download/view file' was triggered by '[email protected]'
> [email protected] start=1459517280810 end=1459517280810
> audits=["AVPR-4oIPeFmuZ3CKKrg","AVPR-wx80cd9PUpAu2aj","AVPR-6XGPeFmuZ3CKKvx","AVPSALn_qE4Kgs_8_yK9","AVPSASW3gw_f3aEvgEmi"]
> services=["APPID_SXC"] users=["[email protected]"]
> cs6=https://abcd-remote.console.arc.com/#/alerts/56fe779ee4b0459f4e9a484a
> cs6Label=consoleUrl
> ...
> {"source.type":"adallom","device_version":"1.0","severity":"0","device_product":"Adallom","services":"[\"APPID_SXC\"]","src_username":"[email protected]","message":"Activity
> policy 'User download\/view file' was triggered by
> '[email protected]'","users":"[\"[email protected]\"]","consoleUrl":"https:\/\/abcd-remote.console.arc.com\/#\/alerts\/56fe779ee4b0459f4e9a484a","event_class_id":"56fe779ee4b0459f4e9a484a","original_string":"2016-04-01T09:29:11.356-0400
>
> CEF:0|Adallom|Adallom|1.0|56fe779ee4b0459f4e9a484a|ALERT_CABINET_EVENT_MATCH_AUDIT|0|msg=Activity
> policy 'User download\/view file' was triggered by '[email protected]'
> [email protected] start=1459517280810 end=1459517280810
> audits=[\"AVPR-4oIPeFmuZ3CKKrg\",\"AVPR-wx80cd9PUpAu2aj\",\"AVPR-6XGPeFmuZ3CKKvx\",\"AVPSALn_qE4Kgs_8_yK9\",\"AVPSASW3gw_f3aEvgEmi\"]
> services=[\"APPID_SXC\"] users=[\"[email protected]\"]
> cs6=https:\/\/abcd-remote.console.arc.com\/#\/alerts\/56fe779ee4b0459f4e9a484a
> cs6Label=consoleUrl","header":"2016-04-01T09:29:11.356-0400
> CEF:0","event_name":"ALERT_CABINET_EVENT_MATCH_AUDIT","startTime":"1459517280810","device_vendor":"Adallom","endTime":"1459517280810","audits":"[\"AVPR-4oIPeFmuZ3CKKrg\",\"AVPR-wx80cd9PUpAu2aj\",\"AVPR-6XGPeFmuZ3CKKvx\",\"AVPSALn_qE4Kgs_8_yK9\",\"AVPSASW3gw_f3aEvgEmi\"]","timestamp":1459502951000}
> CyberArk CEF
> Mar 21 14:05:02 HHHPVATN1 CEF:0|Cyber-Ark|Vault|7.20.0091|295|Retrieve
> password|5|act=Retrieve password suser=spilgrim fname=Root\ABC phobos3 - COMP
> dvc=120.99.70.3 shost=10.44.134.78 dhost= duser= externalId= app= reason=
> cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=Security
> Vulnerability Mgmt cs3Label="Device Type" cs3= cs4Label="Database" cs4=
> cs5Label="Other info" cs5=101.198.70.93 cn1Label="Request Id" cn1=
> cn2Label="Ticket Id" cn2=Needed to verify config files being pulled
> msg=Needed to verify config files being pulled
> ...
> {"timestamp":1458569102000,"source.type":"cyberark","device_version":"7.20.0091","device_product":"Vault","fileName":"Root\\ABC
> phobos3 - COMP","src_username":"spilgrim","\"Other
> info\"":"101.198.70.93","\"Ticket Id\"":"Needed to verify config files being
> pulled
> ","deviceAddress":"120.99.70.3","severity":"5","deviceAction":"Retrieve
> password","message":"Needed to verify config files being
> pulled","event_class_id":"295","original_string":"Mar 21 14:05:02 HHHPVATN1
> CEF:0|Cyber-Ark|Vault|7.20.0091|295|Retrieve password|5|act=Retrieve password
> suser=spilgrim fname=Root\\ABC phobos3 - COMP dvc=120.99.70.3
> shost=10.44.134.78 dhost= duser= externalId= app= reason= cs1Label=\"Affected
> User Name\" cs1= cs2Label=\"Safe Name\" cs2=Security Vulnerability Mgmt
> cs3Label=\"Device Type\" cs3= cs4Label=\"Database\" cs4= cs5Label=\"Other
> info\" cs5=101.198.70.93 cn1Label=\"Request Id\" cn1= cn2Label=\"Ticket Id\"
> cn2=Needed to verify config files being pulled msg=Needed to verify config
> files being pulled","\"Safe Name\"":"Security Vulnerability
> Mgmt","header":"Mar 21 14:05:02 HHHPVATN1 CEF:0","event_name":"Retrieve
> password","device_vendor":"Cyber-Ark","src_hostname":"10.44.134.78"}
> WAF CEF
> <14>CEF:0|Imperva Inc.|SecureSphere|10.0.0.4_16|ABC - Secure Login.vm Page
> Rate Limit UK - Source IP||High|act=alert dst=17.43.200.42 dpt=88
> duser=${Alert.username} src=10.31.45.69 spt=34435 proto=TCP rt=31 March 2016
> 13:04:55 cat=Alert cs1= cs1Label=Policy cs2=ABC-Secure cs2Label=ServerGroup
> cs3=servers_svc cs3Label=ServiceName cs4=server_app cs4Label=ApplicationName
> cs5=QA cs5Label=Description
> ...
> {"source.type":"waf","device_version":"10.0.0.4_16","severity":"High","device_product":"SecureSphere","ServerGroup":"ABC-Secure","ApplicationName":"server_app","Description":"QA","deviceAction":"alert","ip_dst_port":"88","dst_username":"${Alert.username}","priority":"14","deviceEventCategory":"Alert","protocol":"TCP","ip_dst_addr":"17.43.200.42","ip_src_port":"34435","event_class_id":"ABC
> - Secure Login.vm Page Rate Limit UK - Source
> IP","ServiceName":"servers_svc","original_string":"<14>CEF:0|Imperva
> Inc.|SecureSphere|10.0.0.4_16|ABC - Secure Login.vm Page Rate Limit UK -
> Source IP||High|act=alert dst=17.43.200.42 dpt=88 duser=${Alert.username}
> src=10.31.45.69 spt=34435 proto=TCP rt=31 March 2016 13:04:55 cat=Alert cs1=
> cs1Label=Policy cs2=ABC-Secure cs2Label=ServerGroup cs3=servers_svc
> cs3Label=ServiceName cs4=server_app cs4Label=ApplicationName cs5=QA
> cs5Label=Description","header":"<14>CEF:0","device_vendor":"Imperva
> Inc.","ip_src_addr":"10.31.45.69","timestamp":1459429495000}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
