tl;dr - looping in dev mailing list, looking to build sed/grep/cut/pipe
functionality into STELLAR, or looking to take what is available and make it do
this functionality. JIRA is https://issues.apache.org/jira/browse/METRON-933.
this is the intent:
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"winlogbeat",
"parserConfig": {},
"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "TargetUserName","WorkstationName","IpAddress","IpPort" ]
,"config" :
{
"TargetUserName" : "sed 's/,/\r\n/g' original_string |grep "TargetUserName"|cut
-f 2 -d :|sed 's/}//'",
"WorkstationName" : "sed 's/,/\r\n/g' original_string |grep
"WorkstationName"|cut -f 2 -d :|sed 's/}//'",
"IpAddress" : "sed 's/,/\r\n/g' original_string |grep "IpAddress"|cut -f 2 -d
:|sed 's/}//'",
"IpPort" : "sed 's/,/\r\n/g' original_string |grep "IpPort"|cut -f 2 -d :|sed
's/}//'"
}
]
}
________________________________
From: ed d <[email protected]>
Sent: Friday, May 5, 2017 8:28 AM
To: Otto Fowler; [email protected]; [email protected]
Subject: Re: grep sed and cut functionality in STELLAR?
Sure, I will get in the list and post
Get Outlook for Android<https://aka.ms/ghei36>
________________________________
From: Otto Fowler <[email protected]>
Sent: Friday, May 5, 2017 8:27:25 AM
To: [email protected]; [email protected]; ed d
Subject: Re: grep sed and cut functionality in STELLAR?
Can you post this and your ideas to the dev@ list?
On May 5, 2017 at 08:21:29, ed d
([email protected]<mailto:[email protected]>) wrote:
otto, thanks for opening the jira, i was travelling yesterday. im on a mini
vacation until next week but i may take a look and see what is needed to build
out STELLAR, and start experimenting. if we can pipe, then maybe we can cascade
to different variables. i will stab it and see what happens.
________________________________
From: Otto Fowler <[email protected]<mailto:[email protected]>>
Sent: Thursday, May 4, 2017 11:25 PM
To: ed d; [email protected]<mailto:[email protected]>;
[email protected]<mailto:[email protected]>
Subject: Re: grep sed and cut functionality in STELLAR?
I have opened https://issues.apache.org/jira/browse/METRON-933 for
supporting regex capture groups. Not exactly what you need I think, but well,
you gave me the idea :)
On May 4, 2017 at 23:19:19, Otto Fowler
([email protected]<mailto:[email protected]>) wrote:
There are a couple of different ways this could be done:
https://github.com/tools4j/unix4j for example as a back end, with STELLAR
integration and commands ( if the MIT lic is OK ).
Also, there is I believe a Jira for running scripts from stellar that may suit
this.
There is an interesting possibility around using java regex as well.
STELLAR does not from what I can see support unix pipes however.
Do you think you can create a jira for this?
or would you like me to create one for you?
On May 4, 2017 at 16:48:26, ed d
([email protected]<mailto:[email protected]>) wrote:
metron - 0.4.0
parsing windows event logs being pushed from Windows 2012 via winlogbeat 5.3.2
into kafka as JSON
i am parsing the JSON windows event log with the JSONMapParser but there is
vital information in the non-parsed message data portion. i need to extract the
workstation, username, ip address and ip port from the original string as the
JSON parser does not do this.
i think STELLAR can do this but i do not see anything in the STELLAR
documentation
page<https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common><https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common><https://cwiki.apache.org/confluence/display/METRON/Stellar+Query+Language>
to replicate the functionality of "grep", "sed" or "cut", at least i dont
think i do. if it does not exist, i am ready and willing to hack something
together as i understand we can write our own STELLAR functions.
before i apply my non-programming skills to the function, my question to the
group is if anyone has used STELLAR to perform the function of "grep", "sed" or
"cut" on a parser?
below is some background information for this question.
this is what i want the parser to do:
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"winlogbeat",
"parserConfig": {},
"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "TargetUserName","WorkstationName","IpAddress","IpPort" ]
,"config" :
{
"TargetUserName" : "sed 's/,/\r\n/g' original_string |grep "TargetUserName"|cut
-f 2 -d :|sed 's/}//'",
"WorkstationName" : "sed 's/,/\r\n/g' original_string |grep
"WorkstationName"|cut -f 2 -d :|sed 's/}//'",
"IpAddress" : "sed 's/,/\r\n/g' original_string |grep "IpAddress"|cut -f 2 -d
:|sed 's/}//'",
"IpPort" : "sed 's/,/\r\n/g' original_string |grep "IpPort"|cut -f 2 -d :|sed
's/}//'"
}
]
}
this is the data blob:
{"@timestamp":"2017-05-04T11:28:16.171Z","beat":{"hostname":"demometrondc01","name":"demometrondc01","version":"5.3.2"},"computer_name":"demometrondc01.demometron.local","event_data":{"AuthenticationPackageName":"NTLM","ImpersonationLevel":"%%1833","IpAddress":"192.168.56.1","IpPort":"53479","KeyLength":"128","LmPackageName":"NTLM
V2","LogonGuid":"{00000000-0000-0000-0000-000000000000}","LogonProcessName":"NtLmSsp
","LogonType":"3","ProcessId":"0x0","ProcessName":"-","SubjectDomainName":"-","SubjectLogonId":"0x0","SubjectUserName":"-","SubjectUserSid":"S-1-0-0","TargetDomainName":"DEMOMETRON","TargetLogonId":"0x4fed9e","TargetUserName":"ragdelaed","TargetUserSid":"S-1-5-21-2066284954-3896131091-2738420592-1106","TransmittedServices":"-","WorkstationName":"ragdelaed_pc"},"event_id":4624,"keywords":["Audit
Success"],"level":"Information","log_name":"Security","message":"An account
was successfully logged on.\n\nSubject:\n\tSecurity ID:\t\tS-1-0-0\n\tAccount
Name:\t\t-\n\tAccount Domain:\t\t-\n\tLogon ID:\t\t0x0\n\nLogon
Type:\t\t\t3\n\nImpersonation Level:\t\tImpersonation\n\nNew Logon:\n\tSecurity
ID:\t\tS-1-5-21-2066284954-3896131091-2738420592-1106\n\tAccount
Name:\t\tragdelaed\n\tAccount Domain:\t\tDEMOMETRON\n\tLogon
ID:\t\t0x4FED9E\n\tLogonGUID:\t\t{00000000-0000-0000-0000-000000000000}\n\nProcess
Information:\n\tProcess ID:\t\t0x0\n\tProcess Name:\t\t-\n\nNetwork
Information:\n\tWorkstation Name:\tUS1206123W3\n\tSource Network
Address:\t192.168.56.1\n\tSource Port:\t\t53479\n\nDetailed Authentication
Information:\n\tLogon Process:\t\tNtLmSsp \n\tAuthentication
Package:\tNTLM\n\tTransited Services:\t-\n\tPackage Name (NTLM only):\tNTLM
V2\n\tKey Length:\t\t128\n\nThis event is generated when a logon session is
created. It is generated on the computer that was accessed.\n\nThe subject
fields indicate the account on the local system which requested the logon. This
is most commonly a service such as the Server service, or a local process such
as Winlogon.exe or Services.exe.\n\nThe logon type field indicates the kindof
logon that occurred. The most common types are 2 (interactive) and 3
(network).\n\nThe New Logon fields indicate the account for whom the new logon
was created, i.e. the account that was logged on.\n\nThe network fields
indicate where a remote logon request originated. Workstation name is not
always available and may be left blank in some cases.\n\nThe impersonation
level field indicates the extent to which a process in the logon session can
impersonate.\n\nThe authentication information fields provide detailed
information about this specific logon request.\n\t- Logon GUID is a unique
identifier that can be used to correlate this event with a KDC event.\n\t-
Transited services indicate which intermediate services have participated in
this logon request.\n\t- Package name indicates which sub-protocol was used
among the NTLM protocols.\n\t- Key length indicates the length of the generated
session key. This will be 0 if no session keywas
requested.","opcode":"Info","process_id":448,"provider_guid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","record_number":"37123","source_name":"Microsoft-Windows-Security-Auditing","task":"Logon","thread_id":2768,"type":"wineventlog","version":1}
this is how it renders in Metron:
message An account was successfully logged on.
Subject:
Security ID:S-1-0-0
Account Name:-
Account Domain:-
Logon ID:0x0
Logon Type: 3
Impersonation Level: Impersonation
New Logon:
Security ID:S-1-5-21-2066284954-3896131091-2738420592-1106
Account Name:ragdelaed
Account Domain:DEMOMETRON
Logon ID:0x6C7D7C
Logon GUID:{00000000-0000-0000-0000-000000000000}
Process Information:
Process ID:0x0
Process Name:-
Network Information:
Workstation Name:ragdelaed_pc
Source Network Address:192.168.56.1
Source Port:55418
Detailed Authentication Information:
Logon Process:NtLmSsp
Authentication Package:NTLM
Transited Services:-
Package Name (NTLM only):NTLM V2
Key Length:128
