Added: release/metron/0.4.1/site-book/use-cases/geographic_login_outliers/index.html ============================================================================== --- release/metron/0.4.1/site-book/use-cases/geographic_login_outliers/index.html (added) +++ release/metron/0.4.1/site-book/use-cases/geographic_login_outliers/index.html Fri Sep 15 23:37:46 2017 @@ -0,0 +1,532 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-09-08 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170908" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Problem Statement</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Problem Statement</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-09-08</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> + <i class="none"></i> + Alerts</a> + </li> + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + + <li> + + <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li> + + <a href="../../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-down"></i> + Use-cases</a> + <ul class="nav nav-list"> + + <li class="active"> + + <a href="#"><i class="none"></i>Geographic_login_outliers</a> + </li> + </ul> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Problem Statement</h1> +<p><a name="Problem_Statement"></a></p> +<p>One way to find anomalous behavior in a network is by inspecting user login behavior. In particular, if a user is logging in via vastly differing geographic locations in a short period of time, this may be evidence of malicious behavior.</p> +<p>More formally, we can encode this potentially malicious event in terms of how far from the geographic centroid of the user’s historic logins as compared to all users. For instance, if we track all users and the median distance from the central geographic location of all of their logins for the last 2 hours is 3 km and the standard deviation is 1 km, if we see a user logging in 1700 km from the central geographic location of their logins for the last 2 hours, then they MAY be exhibiting a deviation that we want to monitor since it would be hard to travel that distance in 4 hours. On the other hand, the user may have just used a VPN or proxy. Ultimately, this sort of analytic must be considered only one piece of evidence in addition to many others before we want to indicate an alert.</p> +<p><a name="Demonstration_Design"></a></p> +<h1>Demonstration Design</h1> +<p>For the purposes of demonstration, we will construct synthetic data whereby 2 users are logging into a system rather quickly (once per second) from various hosts. Each user’s locations share the same first 2 octets, but will choose the last 2 randomly. We will then inject a data point indicating <tt>user1</tt> is logging in via a russian IP address.</p> +<div class="section"> +<h2><a name="Preliminaries"></a>Preliminaries</h2> +<p>We assume that the following environment variables are set:</p> + +<ul> + +<li><tt>METRON_HOME</tt> - the home directory for metron</li> + +<li><tt>ZOOKEEPER</tt> - The zookeeper quorum (comma separated with port specified: e.g. <tt>node1:2181</tt> for full-dev)</li> + +<li><tt>BROKERLIST</tt> - The Kafka broker list (comma separated with port specified: e.g. <tt>node1:6667</tt> for full-dev)</li> + +<li><tt>ES_HOST</tt> - The elasticsearch master (and port) e.g. <tt>node1:9200</tt> for full-dev.</li> +</ul> +<p>Also, this does not assume that you are using a kerberized cluster. If you are, then the parser start command will adjust slightly to include the security protocol.</p> +<p>Before editing configurations, be sure to pull the configs from zookeeper locally via</p> + +<div class="source"> +<div class="source"> +<pre>$METRON_HOME/bin/zk_load_configs.sh --mode PULL -z $ZOOKEEPER -o $METRON_HOME/config/zookeeper/ -f +</pre></div></div></div> +<div class="section"> +<h2><a name="Configure_the_Profiler"></a>Configure the Profiler</h2> +<p>First, we’ll configure the profiler to emit a profiler every 1 minute:</p> + +<ul> + +<li>In Ambari, set the profiler period duration to <tt>1</tt> minute via the Profiler config section.</li> + +<li>Adjust <tt>$METRON_HOME/config/zookeeper/global.json</tt> to adjust the capture duration:</li> +</ul> + +<div class="source"> +<div class="source"> +<pre> "profiler.client.period.duration" : "1", + "profiler.client.period.duration.units" : "MINUTES" +</pre></div></div></div> +<div class="section"> +<h2><a name="Create_the_Data_Generator"></a>Create the Data Generator</h2> +<p>We want to create a new sensor for our synthetic data called <tt>auth</tt>. To feed it, we need a synthetic data generator. In particular, we want a process which will feed authentication events per second for a set of users where the IPs are randomly chosen, but each user’s login ip addresses share the same first 2 octets.</p> +<p>Edit <tt>~/gen_data.py</tt> and paste the following into it:</p> + +<div class="source"> +<div class="source"> +<pre>#!/usr/bin/python + +import random +import sys +import time + +domains = { 'user1' : '173.90', 'user2' : '156.33' } + +def get_ip(base): + return base + '.' + str(random.randint(1,255)) + '.' + str(random.randint(1, 255)) + +def main(): + freq_s = 1 + while True: + user='user' + str(random.randint(1,len(domains))) + epoch_time = int(time.time()) + ip=get_ip(domains[user]) + print user + ',' + ip + ',' + str(epoch_time) + sys.stdout.flush() + time.sleep(freq_s) + +if __name__ == '__main__': + main() +</pre></div></div></div> +<div class="section"> +<h2><a name="Create_the_auth_Parser"></a>Create the <tt>auth</tt> Parser</h2> +<p>The message format for our simple synthetic data is a CSV with:</p> + +<ul> + +<li>username</li> + +<li>login ip address</li> + +<li>timestamp</li> +</ul> +<p>We will need to parse this via our <tt>CSVParser</tt> and add the geohash of the login ip address.</p> + +<ul> + +<li>To create this parser, edit <tt>$METRON_HOME/config/zookeeper/parsers/auth.json</tt> and paste the following:</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>{ + "parserClassName" : "org.apache.metron.parsers.csv.CSVParser" + ,"sensorTopic" : "auth" + ,"parserConfig" : { + "columns" : { + "user" : 0, + "ip" : 1, + "timestamp" : 2 + } + } + ,"fieldTransformations" : [ + { + "transformation" : "STELLAR" + ,"output" : [ "hash" ] + ,"config" : { + "hash" : "GEOHASH_FROM_LOC(GEO_GET(ip))" + } + } + ] +} +</pre></div></div> + +<ul> + +<li>Create the kafka topic via:</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --create --topic auth --partitions 1 --replication-factor 1 +</pre></div></div></div> +<div class="section"> +<h2><a name="Create_the_Profiles_for_Enrichment"></a>Create the Profiles for Enrichment</h2> +<p>We will need to track 2 profiles to accomplish this task:</p> + +<ul> + +<li><tt>locations_by_user</tt> - The geohashes of the locations the user has logged in from. This is a multiset of geohashes per user. Note that the multiset in this case is effectively a map of geohashes to occurrance counts.</li> + +<li><tt>geo_distribution_from_centroid</tt> - The statistical distribution of the distance between a login location and the geographic centroid of the user’s previous logins from the last 2 minutes. Note, in a real installation this would be a larger temporal lookback.</li> +</ul> +<p>We can represent these in the <tt>$METRON_HOME/config/zookeeper/profiler.json</tt> via the following:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "profiles": [ + { + "profile": "geo_distribution_from_centroid", + "foreach": "'global'", + "onlyif": "exists(geo_distance) && geo_distance != null", + "init" : { + "s": "STATS_INIT()" + }, + "update": { + "s": "STATS_ADD(s, geo_distance)" + }, + "result": "s" + }, + { + "profile": "locations_by_user", + "foreach": "user", + "onlyif": "exists(hash) && hash != null && LENGTH(hash) > 0", + "init" : { + "s": "MULTISET_INIT()" + }, + "update": { + "s": "MULTISET_ADD(s, hash)" + }, + "result": "s" + } + ] +} +</pre></div></div></div> +<div class="section"> +<h2><a name="Enrich_authentication_Events"></a>Enrich authentication Events</h2> +<p>We will need to enrich the authentication records in a couple of ways to use in the threat triage section as well as the profiles:</p> + +<ul> + +<li><tt>geo_distance</tt>: representing the distance between the current geohash and the geographic centroid for the last 2 minutes.</li> + +<li><tt>geo_centroid</tt>: representing the geographic centroid for the last 2 minutes</li> +</ul> +<p>Beyond that, we will need to determine if the authentication event is a geographic outlier by computing the following fields:</p> + +<ul> + +<li><tt>dist_median</tt> : representing the median distance between a user’s login location and the geographic centroid for the last 2 minutes (essentially the median of the <tt>geo_distance</tt> values across all users).</li> + +<li><tt>dist_sd</tt> : representing the standard deviation of the distance between a user’s login location and the geographic centroid for the last 2 minutes (essentially the standard deviation of the <tt>geo_distance</tt> values across all users).</li> + +<li><tt>geo_outlier</tt> : whether <tt>geo_distance</tt> is more than 5 standard deviations from the median across all users.</li> +</ul> +<p>We also want to set up a triage rule associating a score and setting an alert if <tt>geo_outlier</tt> is true. In reality, this would be more complex as this metric is at best circumstantial and would need supporting evidence, but for simplicity we’ll deal with the false positives.</p> + +<ul> + +<li>Edit <tt>$METRON_HOME/config/zookeeper/enrichments/auth.json</tt> and paste the following:</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>{ + "enrichment": { + "fieldMap": { + "stellar" : { + "config" : [ + "geo_locations := MULTISET_MERGE( PROFILE_GET( 'locations_by_user', user, PROFILE_FIXED( 2, 'MINUTES')))", + "geo_centroid := GEOHASH_CENTROID(geo_locations)", + "geo_distance := TO_INTEGER(GEOHASH_DIST(geo_centroid, hash))", + "geo_locations := null" + ] + } + } + ,"fieldToTypeMap": { } + }, + "threatIntel": { + "fieldMap": { + "stellar" : { + "config" : [ + "geo_distance_distr:= STATS_MERGE( PROFILE_GET( 'geo_distribution_from_centroid', 'global', PROFILE_FIXED( 2, 'MINUTES')))", + "dist_median := STATS_PERCENTILE(geo_distance_distr, 50.0)", + "dist_sd := STATS_SD(geo_distance_distr)", + "geo_outlier := ABS(dist_median - geo_distance) >= 5*dist_sd", + "is_alert := exists(is_alert) && is_alert", + "is_alert := is_alert || (geo_outlier != null && geo_outlier == true)", + "geo_distance_distr := null" + ] + } + + }, + "fieldToTypeMap": { }, + "triageConfig" : { + "riskLevelRules" : [ + { + "name" : "Geographic Outlier", + "comment" : "Determine if the user's geographic distance from the centroid of the historic logins is an outlier as compared to all users.", + "rule" : "geo_outlier != null && geo_outlier", + "score" : 10, + "reason" : "FORMAT('user %s has a distance (%d) from the centroid of their last login is 5 std deviations (%f) from the median (%f)', user, geo_distance, dist_sd, dist_median)" + } + ], + "aggregator" : "MAX" + } + } +} +</pre></div></div></div> +<div class="section"> +<h2><a name="Execute_Demonstration"></a>Execute Demonstration</h2> +<p>From here, we’ve set up our configuration and can push the configs:</p> + +<ul> + +<li>Push the configs to zookeeper via</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z node1:2181 -i $METRON_HOME/config/zookeeper/ +</pre></div></div> + +<ul> + +<li>Start the parser via:</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>$METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s auth +</pre></div></div> + +<ul> + +<li>Push synthetic data into the <tt>auth</tt> topic via</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>python ~/gen_data.py | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic auth +</pre></div></div> + +<ul> + +<li>Wait for about <tt>5</tt> minutes and kill the previous command</li> + +<li>Push a synthetic record indicating <tt>user1</tt> has logged in from a russian IP (<tt>109.252.227.173</tt>):</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>echo -e "import time\nprint 'user1,109.252.227.173,'+str(int(time.time()))" | python | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic auth +</pre></div></div> + +<ul> + +<li>Execute the following to search elasticsearch for our geographic login outliers:</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>curl -XPOST "http://$ES_HOST/auth*/_search?pretty" -d ' +{ + "_source" : [ "is_alert", "threat:triage:rules:0:reason", "user", "ip", "geo_distance" ], + "query": { "exists" : { "field" : "threat:triage:rules:0:reason" } } +} +' +</pre></div></div> +<p>You should see, among a few other false positive results, something like the following:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "_index" : "auth_index_2017.09.07.20", + "_type" : "auth_doc", + "_id" : "f5bdbf76-9d78-48cc-b21d-bc434c96e62e", + "_score" : 1.0, + "_source" : { + "geo_distance" : 7879, + "threat:triage:rules:0:reason" : "user user1 has a distance (7879) from the centroid of their last login is 5 std deviations (334.814719) from the median (128.000000)", + "ip" : "109.252.227.173", + "is_alert" : "true", + "user" : "user1" + } +} +</pre></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
Added: release/metron/0.4.1/site-book/use-cases/index.html ============================================================================== --- release/metron/0.4.1/site-book/use-cases/index.html (added) +++ release/metron/0.4.1/site-book/use-cases/index.html Fri Sep 15 23:37:46 2017 @@ -0,0 +1,213 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-09-08 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170908" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Worked Examples</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Worked Examples</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-09-08</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../metron-interface/metron-alerts/index.html" title="Alerts"> + <i class="none"></i> + Alerts</a> + </li> + + <li> + + <a href="../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + + <li> + + <a href="../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li class="active"> + + <a href="#"><i class="icon-chevron-down"></i>Use-cases</a> + <ul class="nav nav-list"> + + <li> + + <a href="../use-cases/geographic_login_outliers/index.html" title="Geographic_login_outliers"> + <i class="none"></i> + Geographic_login_outliers</a> + </li> + </ul> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Worked Examples</h1> +<p><a name="Worked_Examples"></a></p> +<p>The following are worked examples of use-cases that showcase some (or many) component(s) of Metron.</p> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
