Github user markgrover commented on a diff in the pull request:
https://github.com/apache/incubator-spot/pull/7#discussion_r95462618
--- Diff: docs/Open Data Model/Open Data Model.md ---
@@ -0,0 +1,755 @@
+Overview............................................................................
2
+
+Apache Spot Open Data Model
Strategy....................................................................................
2
+
+Apache Spot Enabled Use
Cases...................................................................................................
3
+
+Data
Model.....................................................................................................................................
4
+
+Naming
Convention.......................................................................................................................
5
+
+Prefixes.........................................................................................................................................................
5
+
+Security Event Log/Alert Data
Model..........................................................................................
6
+
+Common..........................................................................................................................................................
7
+
+Network...........................................................................................................................................................
9
+
+File................................................................................................................................................................
10
+
+Endpoint........................................................................................................................................................
11
+
+User...............................................................................................................................................................
11
+
+DNS.............................................................................................................................................................
11
+
+Proxy.........................................................................................................................................................
12
+
+HTTP..............................................................................................................................................................
13
+
+SMTP............................................................................................................................................................
14
+
+FTP.............................................................................................................................................................
15
+
+SNMP....................................................................................................................................................
16
+
+TLS...........................................................................................................................................................
16
+
+SSH...............................................................................................................................................................
17
+
+DHCP.............................................................................................................................................................
17
+
+IRC................................................................................................................................................................
17
+
+Flow............................................................................................................................................................
17
+
+Context
Models............................................................................................................................
18
+
+User Context
Model.......................................................................................................................
18
+
+Endpoint Context
Model..........................................................................................................................
20
+
+Network Context
Model............................................................................................................................22
+
+Extensibility of Data
Model.........................................................................................................
23
+
+Model
Relationships....................................................................................................................
24
+
+Data Ingestion
Framework..........................................................................................................
24
+
+Data
Formats................................................................................................................................
25
+
+Avro...............................................................................................................................................................
25
+
+JSON......................................................................................................................................................
27
+
+Parquet...................................................................................................................................................
27
+
+ODM Resultant Capability - A Singular
View............................................................................
28
+
+**Example - Advanced Threat
Modeling**...................................................................................................
28
+
+**Example - Singular Data View for Complete
Context**................................................................. 29
+
+
+
+**Overview**
+----
+
+This document describes a strategy for creating an open data model (ODM)
for Apache Spot (incubating) (formerly known as âOpen Network Insight
(ONI)â) in support of cyber security analytic use cases. It also describes
the use cases for which Apache Spot (incubating) running on the Cloudera
platform is uniquely capable of addressing along with the data model.
+
+
+
+**Apache Spot (incubating) Open Data Model Strategy**
+------------------------------------
+
+The Apache Spot (incubating) Open Data Model (ODM) strategy aims to extend
Apache Spot (incubating) capabilities to support a broader set of cyber
security use cases than initially supported. The primary use case initially
supported by Apache Spot (incubating) includes Network Traffic Analysis for
network flows (Netflow, sflow, etc.), DNS and Proxy; primarily the
identification of threats through anomalous event detection using both
supervised and unsupervised machine learning.
+
+In order to support a broader set of use cases, Spot must be extended to
collect and analyze other common
+âevent-orientedâ data sources analyzed for cyber threats, including
but not limited to the following log types:
+
+> âProxy
+>
+> âWeb server
+>
+> âOperating system
+>
+> âFirewall
+>
+> âIntrusion Prevention/Detection (IDS/ IPS)
+>
+> âData Loss Prevention
+>
+> âActive Directory / Identity Management
+>
+> âUser/Entity Behavior Analysis
+>
+> âEndpoint Protection/Asset Management
+>
+> âNetwork Metadata/Session and PCAP files
+>
+> âNetwork Access Control
+>
+> âMail
+>
+> âVPN
+>
+> â etc..
+
+One of the biggest challenges organizations face today in combating cyber
threats is collecting and normalizing data from the myriad of security event
data sources (hundreds) in order to build the needed analytics. This often
results in the analytics being dependent upon the specific technologies used by
an organization to detect threats and prevents the needed flexibility and
agility to keep up with these ever-increasing (and complex) threats.
Technology lock-in is sometimes a byproduct of todayâs status quo, as itâs
extremely costly to add new technologies (or replace existing ones) because of
the downstream analytic dependencies.
+
+To achieve the goal of extending Apache Spot (incubating) to support
additional use cases, it is necessary to create an open data model for the most
relevant security event and contextual data sources; Security event logs or
alerts, Network context, User details and information that comes from the
endpoints or any other console that are being use to manage the security /
administration of our endpoints. The presence of an open data model, which can
be applied âon-readâ or âon-writeâ, in batch or stream, will allow for
the separation of security analytics from the specific data sources on which
they are built. This âseparation of dutiesâ will enable organizations to
build analytics that are not dependent upon specific technologies and provide
the flexibility to change underlying data sources and also provide segmentation
of this information, without impacting the analytics. This will also afford
security vendors the opportunity to build additional products on top of t
he Open Data Model to drive new revenue streams and also to design new ways to
detect threats and APT.
+
+
+**Apache Spot (incubating) Enabled**
+----
+
+**Use Cases**
+-------------
+
+Spot on the Cloudera platform is uniquely positioned to help address the
following cyber security use cases,
+which are not effectively addressed by legacy technologies:
+
+
+
+ **- Detection of known & unknown threats leveraging machine learning and
advanced analytic modeling**
+
+Current technologies are limited in the analytics they can apply to detect
threats. These limitations stem from the inability to collect all the data
sources needed to effectively identify threats (structured, unstructured, etc.)
and inability to process the massive volumes of data needed to do so (billions
of events per day). Legacy technologies are typically focus and limited to
rules-based and signature detection. They are somewhat âeffectiveâ at
detecting known threats but struggle with new threats.
+
+Spot addresses these gaps through its ability to collect any data type of
any volume. Coupled with the various analytic frameworks that are provided
(including machine learning), Spot enables a whole new class of analytics that
can scale to todayâs demands. The topic model used by Spot to detect
anomalous network traffic is one example of where the Spot platform excels.
+
+ **- Reduction of mean time to incident detection & resolution (MTTR)**
+
+One of the challenges organizations face today is detecting threats early
enough to minimize adverse impacts. This stems from the limitations previously
discussed with regards to limited analytics. It can also be attributed to the
fact that most of the investigative queries often take hours or days to return
results. Legacy technologies canât offer or have a central data store for
facilitating such investigations due to their inability to store and serve the
massive amounts of data involved. This cripples incident investigations and
results in MTTRs of many weeks or months, meanwhile the adverse impacts of the
breach are magnified, thus making the threat harder to eradicate.
+
+Apache Spot (incubating) addresses these gaps by providing the capability
for a central data store that houses ALL the data needed to facilitate an
investigation, returning investigative query results in seconds and minutes
(vs. hours and days). Spot can effectively reduce incident MTTR and reduce
adverse impacts of a breach.
+
+ **- Threat Hunting**
+
+Itâs become necessary for organizations to âhuntâ for active threats
as traditional passive threat detection approaches are not sufficient.
âHuntingâ involves performing ad-hoc searches and queries over vast amounts
of data representing many weeks and monthsâ worth of events, as well as
applying ad-hoc / tune algorithms to detect the needle in the haystack.
Traditional systems do not perform well for these types of activities as the
query results sometimes take hours and days to be retrieved. These traditional
systems also lack the analytic flexibility to construct the necessary
algorithms and logic needed.
+
+Apache Spot (incubating) addresses these gaps in the same ways it
addresses others; by providing a central data store with the needed analytic
frameworks that scale to the needed workloads.
+
+**Data Model**
+----------
+In order to provide a framework for effectively analyzing data for cyber
threats, it is necessary to collect and
+analyze standard security event logs/alerts and contextual data regarding
the entities referenced in these logs/alerts. The most common entities include
network, user and endpoint, but there are others such as file.
+
+In the diagram below, the raw event tells us that user âjsmithâ
successfully logged in to an Oracle database from the IP address 10:1.1.3.
Based on the raw event only, we donât know if this event is a legitimate
threat or not. After injecting user and endpoint context, the enriched event
tells us this event is a potential threat that requires further investigation.
+
+
+
+Based on the need to collect and analyze both security events, logs or
alerts and contextual data, support for
+the following types of security information are planned for inclusion in
the Spot Open Data Model:
+
+ - Security event logs/alerts
+This data type includes event logs from common data sources used to detect
threats and includes network flows, operating system logs, IPS/IDS logs,
firewall logs, proxy logs, web logs, DLP logs, etc.
+
+ - Network context data
+This data type includes information about the network, which can be
gleaned from Whois servers, asset databases and other similar data sources.
+
+ - User context data
+This data type includes information from user and identity management
systems including Active Directory, Centrify, and other identity and access
management systems.
+
+ - Endpoint context data
+This data includes information about endpoint systems (servers,
workstations, routers, switches, etc.) and can be sourced from asset management
systems, vulnerability scanners, and endpoint management/detection/response
systems such as Webroot, Tanium, Sophos, Endgame, CarbonBlack, Intel Security
ePO and others.
+
+ - File context data** (ROADMAP ITEM)**
+This data includes contextual information about files and can be sourced
from systems such as FireEye, Application Control and others.
+
+ - Threat intelligence context data **(ROADMAP ITEM)**
+This data includes contextual information about URLs, domains, websites,
files and others.
+
+**Naming Convention**
+-----------------
+
+A naming convention is needed for the Open Data Model to represent common
attributes across vendor products and technologies. The naming convention is
described below.
+
+**Prefixes**
+--------
+
+| Prefix | Description |
+|---|---|
+| src | Corresponds to the âsourceâ fields within a given event (i.e.
source address)|
+| dst | Corresponds to the âdestinationâ fields within a given event
(i.e. destination address) |
+| dvc | Corresponds to the âdeviceâ applicable fields within a given
event (i.e. device address) and represent where the event originated |
+| fwd | Forwarded from device |
+| request | Corresponds to requested values (vs. those returned, i.e.
ârequested URIâ) |
+| response | Corresponds to response value (vs. those requested) |
+| file | Corresponds to the âfileâ fields within a given event (i.e.
file type) |
+| user | Corresponds to user attributes (i.e. name, id, etc.) |
+| xlate | Corresponds to translated values within a given event (i.e.
src_xlate_ip for âtranslated source ip addressâ |
+| in | Ingress|
+| out | Egress |
+| new | New value |
+| orig | Original value |
+| app | Corresponds to values associated with application events |
+
+
+**Security Event Log/Alert Data Model**
+-----------------------------------
+
+The data model for security event logs/alerts is detailed in the below.
The attributes are categorized as follows:
+
+ - Common -attributes that are common across many device types
+ - Device -attributes that are applicable to the device that generated the
event
+ - File -attributes that are applicable to file objects referenced in the
event
+ - Endpoint -attributes that are applicable to the endpoints referenced in
the event
+ - User- attributes that are applicable to the user referenced in the event
+ - Proxy - attributes that are applicable to proxy events
+ - Protocol
+
+> DNS - attributes that are specific to DNS events
+> HTTP - attributes that are specific to HTTP events
+> SMTP, SSH, TLS, DHCP, IRC, SNMP and FTP
+
+Note: The model will evolve to include reserved attributes for additional
device types that are not currently represented. The model can currently be
extended to support ANY attribute for ANY device type by following the guidance
outlined in the section titled **âExtensibility of Data Modelâ.**
+
+Note: Attributes denoted in BLUE represent those that are listed in the
model multiple times for the purpose of
+demonstrating attribute coverage for a particular entity (endpoint, user,
network, etc.) or log type (Proxy, DNS, etc.).
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|***Common***|eventtime|long|timestamp of event (UTC)|1472653952|
+||duration|int|Time duration (milliseconds)|2345|
+||eventid|string|Unique identifier for event|x:2388|
+||org|string|Organization|âHRâ or âFinanceâ or âCustomerAâ
+||type|string|Type information |âInformationalâ, âimage/gifâ
+||nproto|string|Network protocol of event |TCP, UDP, ICMP
+||aproto|string|Application protocol of event |HTTP, NFS, FTP
+||msg|string|Message (details of action taken on object)|Some long string
+||mac|string|MAC address|94:94:26:3:86:16
+||severity|string|Severity of event|High, 10, 1
+||raw|string|Raw text message of entire event|Complete copy of log entry
+||risk|Floating point|Risk score|95.67
+||code|string|Response or error code|404
+||category|string|Event category|/Application/Start
+||qry|string|Query (DNS query, URI query, SQL query, etc.)|Select * from
"table"
+||service|string|(i.e. service name, type of service)|sshd
+||state|string|State of object|Running, Paused, stopped
+||in_bytes|int|Bytes in|1025
+||out_bytes|int|Bytes out|9344
+||additional_attrs|String (JSON Map)|Custom event
attributes|"building":"729","cube":"401"|
+||dvc_time|long|UTC timestamp from device where event/alert originates or
is received|1472653952|
+||dvc_ip4/dvc_ip6|long|IP address of device|Integer representaion of
10.1.1.1|
+||dvc_host|string|Hostname of device|Integer representaion of 10.1.1.1|
+||dvc_type|string|Device type that generated the log|Unix, Windows,
Sonicwall|
+||dvc_vendor|string|Vendor|Microsoft, Fireeye, Intel Security|
+||dvc_version|string|Version |5.4|
+||fwd_ip4/fwd_ip6|long|Forwarded from device|Integer representation of
10.1.1.1|
+||version|string|Version|â3.2.2â|
+
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**Network**|src_ip4/src_ip6|bigint|Source ip address of event|Integer
representation of 10.1.1.1
+||src_host|string|Source FQDN of event|test.companyA.com
+||src_domain|string|Domain name of source address|companyA.com
+||src_port|int|Source port of event|1025
+||src_country_code|string|Source country code|cn
+||src_country_name|string|Source country name|China
+||src_region|string|Source region|string
+||src_city|string|Source city|Shenghai
+||src_lat|int|Source latitude|
+||src_long|int|Source longitude|
+||dst_ip4/dst_ip6|bigint|Destination ip address of event|Integer
representaion of 10.1.1.1
+||dst_host|string|Destination FQDN of event|test.companyA.com
+||dst_domain|string|Domain name of destination address|companyA.com
+||dst_port|int|Destination port of event|80
+||dst_country_code|string|Source country code|cn
+||dst_country_name|string|Source country name|China
+||dst_region|string|Source region|string
+||dst_city|string|Source city|Shenghai
+||dst_lat|int|Source latitude|
+||dst_long|int|Source longitude|
+||asn|int|Autonomous system number|33
+||in_bytes|int|Bytes in|987
+||out_bytes|int|Bytes out|1222
+||direction|string|Direction|In, inbound, outbound, ingress, egress
+||flags|string|TCP flags|.AP.SF
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**File**|file_name|string|Filename from event|output.csv
+||file_path|string|File path|/root/output.csv
+||file_atime|bigint|Timestamp (UTC) of file access|1472653952
+||file_acls|string|File permissions|rwx-rwx-rwx
+||file_type|string|Type of file|â.docâ
+||file_size|int|Size of file in bytes|1244
+||file_desc|string|Description of file|Project Plan for Project xyz
+||file_hash|string|Hash of file|
+||file_hash_type|string|Type of hash|MD5, SHA1,SHA256
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**Endpoint**|object|string|File/Process/Registry|File, Registry, Process
+||action|string|Action taken on object (open/delete/edit)|Open, Edit
+||msg|string|Message (details of action taken on object)|Some long string
+||app|string|Application|Microsoft Powerpoint
+||location|string|Location|Atlanta, GA
+||proc|string|Process|SSHD
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**User**|user_name|string|username from event|mhicks
+||email|string|Email address|[email protected]
+||user_id|string|userid|234456
+||user_loc|string|location|Herndon, VA
+||user_desc|string|Description of user|
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|DNS|dns_class|string|DNS class|1
+||dns_length|int|DNS frame length|188
+||dns_qry|string|Requested DNS query|test.test.com
+||dns_code|string|Response code|0x00000001
+||dns_response_qry|string|Response to DNS Query|178.2.1.99
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|Proxy|category|string|Event category|SG-HTTP-SERVICE
+||browser|string|Web browser|Internet Explorer
+||code|string|Error or response code|404
+||in_bytes|int|Bytes in|1025
+||out_bytes|int|Bytes out|1288
+||referrer|string|Referrer|www.usatoday.com
+||request_uri|string|Requested URI|/wcm/assets/images/imagefileicon.gif
+||filter_rule|string|Applied filter or rule|Internet, Rule 6
+||filter_result|string|Result of applied filter or rule|Proxied, Blocked
+||qry|string|URI query|?func=S_senseHTML&Page=a26815a313504697a126279
+||action|string|Action taken on object |TCP_HIT, TCP_MISS, TCP_TUNNELED
+||method|string|HTTP method|GET, CONNECT, POST
+||type|string|Type of request|image/gif
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|HTTP|request_method|string|HTTP method|GET, CONNECT, POST
+||request_uri |string|Requested URI|/wcm/assets/images/imagefileicon.gif
+||request_body_len|int|Length of request body|98
+||request_user_name |string|username from event|mhicks
+||request_password|string|Password from event|abc123
+||request_proxied|string||
+||request_headers|MAP|HTTP request headers|request_headers[âHOSTâ]
request_headers[âUSER-AGENTâ] request_headers[âACCEPTâ]
+||response_status_code|int|HTTP response status code|404
+||response_status_msg|string|HTTP response status message|âNot foundâ
+||response_body_len|int|Length of response body|98
+||response_info_code |int|HTTP response info code|100
+||response_info_msg|string|HTTP response info message|âSome stringâ
+||response_resp_fuids|string|Response FUIDS|
+||response_mime_types|string|Mime types|âcgi,bat,exeâ
+||response_headers|MAP|Response headers|response_headers[âSERVERâ]
response_headers[âSET-COOKIEââ] response_headers[âDATEâ]
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**SMTP**|trans_depth|int|Depth of email into SMTP exchange|Coming soon
+||headers_helo|string|Helo header|Coming soon
+||headers_mailfrom|string|Mailfrom header|Coming soon
+||headers_rcptto|string|Rcptto header|Coming soon
+||headers_date|string|Header date|Coming soon
+||headers_from|string|From header|Coming soon
+||headers_to|string|To header|Coming soon
+||headers_reply_to|string|Reply to header|Coming soon
+||headers_msg_id|string|Message ID |Coming soon
+||headers_in_reply_to|string|In reply to header|Coming soon
+||headers_subject|string|Subject|Coming soon
+||headers_x_originating_ip4|bigint|Originating IP address|Coming soon
+||headers_first_received|string|First to receive message|Coming soon
+||headers_second_received|string|Second to receive message|Coming soon
+||last_reply|string|Last reply in message chain|Coming soon
+||path|string|Path of message|Coming soon
+||user_agent|string|User agent|Coming soon
+||tls|boolean|Indication of TLS use|Coming soon
+||is_webmail|boolean|Indication of webmail|Coming soon
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**FTP**|user_name|string|Username|Coming soon
+||password|string|Password|Coming soon
+||command|string|FTP command|Coming soon
+||arg|string|Argument|Coming soon
+||mime_type|string|Mime type|Coming soon
+||file_size|int|File size|Coming soon
+||reply_code|int|Reply code|Coming soon
+||reply_msg|string|Reply message|Coming soon
+||data_channel_passive|boolean|Passive data channel?|Coming soon
+||data_channel_rsp_p|string||Coming soon
+||cwd|string|Current working directory|Coming soon
+||cmdarg_ts|float||Coming soon
+||cmdarg_cmd|string|Command|Coming soon
+|cmdarg_arg|string|Command argument|Coming soon
+||cmdarg_seq|int|Sequence|Coming soon
+||pending_commands|string|Pending commands|Coming soon
+||is_passive|boolean|Passive mode enabled|Coming soon
+||fuid|string|Coming soon|Coming soon
+||last_auth_requested|string|Coming soon|Coming soon
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**SNMP**|version|string|Coming soon|Coming soon
+||community|string|Coming soon|Coming soon
+||get_requests|int|Coming soon|Coming soon
+||get_bulk_requests|int|Coming soon|Coming soon
+||get_responses|int|Coming soon|Coming soon
+||set_requests|int|Coming soon|Coming soon
+||display_string|string|Coming soon|Coming soon
+||up_since|float|Coming soon|Coming soon
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**TLS**|version|string|Coming soon|Coming soon
+||cipher|string|Coming soon|Coming soon
+||curve|string|Coming soon|Coming soon
+||server_name|string|Coming soon|Coming soon
+||resumed|boolean|Coming soon|Coming soon
+||next_protocol|string|Coming soon|Coming soon
+||established|boolean|Coming soon|Coming soon
+||cert_chain_fuids|string|Coming soon|Coming soon
+||client_cert_chain_fuids|string|Coming soon|Coming soon
+||subject|string|Coming soon|Coming soon
+||issuer|string|Coming soon|Coming soon
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**SSH**|version|string|Coming soon|Coming soon
+||auth_success|boolean|Coming soon|Coming soon
+||client|string|Coming soon|Coming soon
+||server|string|Coming soon|Coming soon
+||cipher_algorithm|string|Coming soon|Coming soon
+||mac_algorithm|string|Coming soon|Coming soon
+||compression_algorithm|string|Coming soon|Coming soon
+||key_exchange_algorithm|string|Coming soon|Coming soon
+||host_key_algorithm|string|Coming soon|Coming soon
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**DHCP**|assigned_ip4|bigint|Coming soon|Coming soon
+||mac|string|Coming soon|Coming soon
+||lease_time|double|Coming soon|Coming soon
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**IRC**|user|string|Coming soon|Coming soon
+||nickname|string|Coming soon|Coming soon
+||command|string|Coming soon|Coming soon
+||value|string|Coming soon|Coming soon
+||additional_data|string|Coming soon|Coming soon
+
+|**Category**|**Attribute**|**Data Type**|**Description**|**Sample
Values**|
+|---|---|---|---|---|
+|**Flow**|in_packets|int|Coming soon|Coming soon
+||out_packets|int|Coming soon|Coming soon
+||in_bytes|int|Coming soon|Coming soon
+||out_bytes|int|Coming soon|Coming soon
+||conn_state|string|Coming soon|Coming soon
+||history|string|Coming soon|Coming soon
+||duration|float|Coming soon|Coming soon
+||src_os|string|Coming soon|Coming soon
+||dst_os|string|Coming soon|Coming soon
+
+Note: It is not necessary to populate all of the attributes within the
model. For attributes not populated in a single security event log/alert,
contextual data may not be available. For example, the sample event below can
be enriched with contextual data about the referenced endpoints (10.1.1.1 and
192.168.10.10), but not a user, because username is not populated.
+
+>
**date,time,source_ip,source_port,protocol,destination_ip,destination_port,bytes**
+12/12/2015,23:14:56,10.1.1.1,1025,tcp,192.168.10.10,443,1183
+
+
+**Context Models**
+==================
+The recommended approach for populating the context models (user,
endpoint, network, etc.) involves consuming information from the systems most
capable or providing the needed context. Populating the user context model is
best accomplished by leveraging user/identity management systems such as Active
Directory or Centrify and populating the model with details such as the
userâs full name, job title, phone number, managerâs name, physical
address, entitlements, etc. Similarly, an endpoint model can be populated by
consuming information from endpoint/asset management systems (Tanium, Webroot,
etc.), which provide information such as the services running on the system,
system owner, business context, etc.
+
+**User Context Model**
+------------------
+The data model for user context information is as follows:
+
+|**Attribute**|**Data Type**|**Description**|**Sample Values**|
+|---|---|---|---|
+|dvc_time|bigint|Timestamp from when the user context information is
obtained|1472653952
+|created|bigint|Timestamp from when user was created|1472653952
+|Changedââââ|bigint|Timestamp from when user was updated|1472653952
+|lastlogon|bigint|Timestamp from when user last logged on|1472653952
+|logoncount|int|Number of times account has logged on|232
+|lastreset|bigint|Timestamp from when user last reset passwod|1472653952
+|expiration|bigint|Date/time when user expires|1472653952
+|userid|string|Unique user id|1234
+|username|string|Username in event log/alert|jsmith
+|name_first|string|First name|John
+|name_middle|string|Middle name|Henry
+|name_last|string|Last name|Smith
+|name_mgr|string|Managerâs name|Ronald Reagan
+|phone|string|Phone number|703-555-1212
+|email|string|Email address|[email protected]
+|code|string|Job code|3455
+|loc|string|Location|US
+|departm|string|Department|IT
+|dn||Distinguished
name|"CN=scm-admin-mej-test2-adk,OU=app-|admins,DC=ad,DC=halxg,DC=companya,DC=com"
+|ou|string|Organizational unit|EAST
+|empid|string|Employee ID|12345
+|title|string|Job Title|Director of IT
+|groups|string (comma separated list, no spaces after comma)|Groups to
which the user belongs|âDomain Adminsâ, âDomain Usersâ
+|dvc_type|string|Device type that generated the user context data|Active
Directory
+|dvc_vendor|string|Vendor|Microsoft
+|dvc_version|string|Version |8.1.2
+|additional_attrs|string|Additional attributes of user|Key value pairs
+
+
+**Endpoint Context Model**
+------------------
+The data model for endpoint context information is as follows:
+|**Abbreviation**|**Data Type**|**Description**|**Sample Values**|
+|---|---|---|---|
+|dvc_time|bigint|Timestamp from when the endpoint context information is
obtained|1472653952
+|ip4|bigint|IP address of endpoint |Integer representaion of 10.1.1.1
+|ip6|bigint|IP address of endpoint |Integer representaion of 10.1.1.1
+|os|string|Operating system|Redhat Linux 6.5.1
+|os_version|string|Version of OS|5.4
+|os_sp|string|Service pack|SP 2.3.4.55
+|tz|string|timezone|EST
+|hotfixes|string|Applied hotfixes|993.2
+|disks|string|Available disks|\\Device\\HarddiskVolume1,
\\Device\\HarddiskVolume2
+|removables|string|Removable media devices|USB Key
+|nics|string|Network interfaces|fe10::28f4:1a47:658b:d6e8,
fe82::28f4:1a47:658b:d6e8
+|drivers|string|Installed kernel drivers|ntoskrnl.exe, hal.dll
+|users|string|Local user accounts|administrator, jsmith
+|host|string|Hostname of endpoint|tes1.companya.com
+|mac|string|MAC address of endpoint|fe10::28f4:1a47:658b:d6e8
+|owner|string|Endpoint owner (name)|John Smith
+|vulns|string (comma separated, no spaces after commas)|Vulnerability
identifiers (CVE identifier)|CVE-123, CVE-456
+|loc|string|Location|US
+|departm|string|Department name|IT
+|company|string|Company name|CompanyA
+|regs|string (comma-separated)|Applicable regulations|HIPAA, SOX
+|svcs|string (comma-separated)|Services running on system|Cisco Systems,
Inc. VPN Service, Adobe LM Service
+|procs|string|Processes|svchost.exe, sppsvc.exe
+|criticality|string|Criticality of device|Very High
+|apps|string (comma-separated)|Applications running on system|Microsoft
Word, Chrome
+|desc|string|Endpoint descriptor|Some string
+|dvc_type|string|Device type that generated the log|Microsoft Windows 7
+|dvc_vendor|string|Vendor|Endgame
+|dvc_version|string|Version |2.1
+|architecture|string|CPU architecture|x86
+|uuid|string|Universally unique
identifier|a59ba71e-18b0-f762-2f02-0deaf95076c6
+|memtotal|int|Total memory (bytes)|844564433
+|additional_attrs|string|Additional attributes|Key value pairs
+
+**VPN Context Model**
+------------------
+The data model for VPN context information is based on the VPN logs as
follows:
+
+|**Abbreviation**|**Data Type**|**Description**|**Sample Values**|
+|---|---|---|---|
+|dvc_time|bigint|Timestamp from when the endpoint context information is
obtained|1472653952
+|ip4|bigint|IP address of VPN box|Integer representaion of 10.1.1.1
+|ip6|bigint|IP address of VPN box |Integer representaion of 10.1.1.1
+|vpn_vendor|string|Vendor VPN|Cisco
+|vpn_version|string|Version VPN|3.0
+|vpn_sp|string|VPN Service pack|5
+|tz|string|VPN timezone|EST
+|vpn_hotfixes|string|VPN Applied hotfixes|1134
+|vpn_nics|string|Network interfaces|fe10::28f4:1a47:658b:d6e8,
fe82::28f4:1a47:658b:d6e8
+|vpn_host|VPN Country Code|string|MX
+|vpn_country_name|VPN Country Name|string|Mexico
+|vpn_ip||string|Integer representation of 10.1.1.2
+|vpn_encrypt|VPN encryption protocol|string|IPSEC
+|vpn_username|string|VPN user account|jsmith
+|vpn_user_ip|string|VPN User IP address|Integer representation of 10.1.1.2
+|vpn_user_cc|string|VPN Country Code|US
+|vpn_user_cn|string|VPN Country Name|United States
+|vpn_user_auth|string|VPN user authorization / role|Admin, normal user, etc
+|vpn_account_vip|string|Criticality of the VPN account|Medium, High
+|vpn_uuid|string|Universally unique
identifier|a59ba71e-18b0-f762-2f02-0deaf95076c6
+|uuids|string|Universally unique identifier(s) comes from thee endpoint
context if match|a59ba71e-18b0-f762-2f02-0deaf95xmexzA
+|additional_attrs|string|Additional attributes|Key value pairs
+
+**Network Context Model**
+------------------
+The data model for network context information is based on âwhoisâ
information as follows:
+
+|**Attribute**|**Data Type**|**Description**|**Sample Values**|
+|---|---|---|---|
+|domain_name|string|Domain name
+|registry_domain_id|string|Registry Domain ID
+|registrar_whois_server|string|Registrar WHOIS Server
+|registrar_url|string|Registrar URL
+|update_date|bigint|UTC timestamp
+|creation_date|bigint|Creation Date
+|registrar_registration_expiration_date|bigint|Registrar Registration
Expiration Date
+|registrar|string|Registrar
+|registrar_iana_id|string|Registrar IANA ID
+|registrar_abuse_contact_email|string|Registrar Abuse Contact Email
+|registrar_abuse_contact_phone|string|Registrar Abuse Contact Phone
+|domain_status|string|Domain Status
+|registry_registrant_id|string|Registry Registrant ID
+|registrant_name|string|Registrant Name
+|registrant_organization|string|Registrant Organization
+|registrant_street|string|Registrant Street
+|registrant_city|string|Registrant City
+|registrant_state_province|string|Registrant State/Province
+|registrant_postal_code|string|Registrant Postal Code
+|registrant_country|string|Registrant Country
+|registrant_phone|string|Registrant Phone
+|registrant_email|string|Registrant Email
+|registry_admin_id|string|Registry Admin ID
+|name_server|string|Name Server
+|dnssec|string|DNSSEC
+
+
+**Extensibility of Data Model**
+==================
+
+The aforementioned data model can be extended to accommodate custom
attributes by embedding key-value pairs within the log/alert/context entries.
+Each model will support an additional attribute by the name of
additional_attrs whose value would be a JSON string. This JSON string will
contain a Map (and only a Map) of additional attributes that canât be
expressed in the specified model description. Regardless of the type of these
additional attributes, they will always be interpreted as String. Itâs up to
the user, to translate them to appropriate types, if necessary, in the
analytics layer. It is also the userâs responsibility to populate the
aforementioned attribute as a Map, by presumably parsing out these attributes
from the original message.
+For example, if a user wanted to extend the user context model to include
a string attribute for âDesk Locationâ and âCityâ, the following string
would be set for additional_attrs:
+
+|**Attribute Key**|**Attribute Value**|
+|---|---|
+|additional_attrs|{"dsk_location":"B3-F2-W3", "city":"Palo Alto"}|
+
+
+Something similar can be done for endpoint context model, security event
log/alert model and other entities.
+
+ Note: This [UDF library](https://github.com/klout/brickhouse) can be
used for converting to/from JSON.
+
+##**Model Relationships**##
+
+The relationships between the data model entities are illustrated below.
+
+Image here
+
+
+##**Data Ingestion Framework**##
+
+One of the challenges in populating the data model is the large number of
products and technologies that organizations are currently using to manage
security event logs/alerts, user and endpoint information. There are literally
dozens of vendors in each category that offer technologies that could be used
to populate the model. The labor required to transform the data and map the
attributes to the data model is extensive when you consider how many
technologies are in the mix at each organization (and across organizations).
One way to address this challenge is with a Data Ingestion Framework that
provides a configuration-based mechanism to perform the transformations and
mappings. A configuration-based capability will allow the ingest pipelines to
become portable and reusable across the community. For example, if I create an
ingest pipeline for Centrify to populate the user context model, it can be
shared with other users of Centrify who can immediately realize the benefit.
Suc
h a framework could allow the community to quickly build the necessary
pipelines for the dozens (and hundreds) of technologies being used in the
market. Without a standard ingest framework, each pipeline is built
independently, requiring more labor, providing no standardization and little
portability. Itâs also important that the data ingestion framework support
the ability to both capture the ârawâ event and create a meta event that
represents the normalized event and maps the attributes to the defined data
model. This will ensure both stream and batch processing use cases are
supported.
+
+Streamsets is an ingest framework that provides the needed functionality
outlined above. Sample Streamsets ingest pipelines for populating the ODM with
common data sources will be published to the Spot Github repo.
+
+##**Data Formats**##
+
+**Avro**
+----
+
+Avro is the recommended data format due to its schema representation,
compatibility checks, and interoperability with Hadoop. Avro supports a pure
JSON representation for readability and ease of use but also a binary
representation of the data for efficient storage. Avro is the optimal format
for streaming-based analytic use cases.
+
+A sample event and corresponding schema representation are detailed below.
+
+**Event**
--- End diff --
This may get compacted in one line so you have to put 3 backticks at top
and bottom of this block. Same for Schema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
