Added: release/metron/0.4.0/site-book/metron-analytics/metron-statistics/index.html ============================================================================== --- release/metron/0.4.0/site-book/metron-analytics/metron-statistics/index.html (added) +++ release/metron/0.4.0/site-book/metron-analytics/metron-statistics/index.html Wed Jul 5 06:56:42 2017 @@ -0,0 +1,943 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-06-27 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Statistics and Mathematical Functions</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Statistics and Mathematical Functions</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-down"></i> + Analytics</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service"> + <i class="none"></i> + Maas-service</a> + </li> + + <li> + + <a href="../../metron-analytics/metron-profiler/index.html" title="Profiler"> + <i class="none"></i> + Profiler</a> + </li> + + <li> + + <a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"> + <i class="none"></i> + Profiler-client</a> + </li> + + <li class="active"> + + <a href="#"><i class="icon-chevron-down"></i>Statistics</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-analytics/metron-statistics/HLLP.html" title="HLLP"> + <i class="none"></i> + HLLP</a> + </li> + </ul> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Statistics and Mathematical Functions</h1> +<p><a name="Statistics_and_Mathematical_Functions"></a></p> +<p>A variety of non-trivial and advanced analytics make use of statistics and advanced mathematical functions. Particular, capturing the statistical snapshots in a scalable way can open up doors for more advanced analytics such as outlier analysis. As such, this project is aimed at capturing a robust set of statistical functions and statistical-based algorithms in the form of Stellar functions. These functions can be used from everywhere where Stellar is used.</p> +<div class="section"> +<h2><a name="Stellar_Functions"></a>Stellar Functions</h2> +<div class="section"> +<h3><a name="Approximation_Statistics"></a>Approximation Statistics</h3> +<div class="section"> +<h4><a name="HLLP_ADD"></a><tt>HLLP_ADD</tt></h4> + +<ul> + +<li>Description: Add value to the HyperLogLogPlus estimator set. See <a href="HLLP.html">HLLP README</a></li> + +<li>Input: + +<ul> + +<li>hyperLogLogPlus - the hllp estimator to add a value to</li> + +<li>value+ - value to add to the set. Takes a single item or a list.</li> + </ul></li> + +<li>Returns: The HyperLogLogPlus set with a new value added</li> +</ul></div> +<div class="section"> +<h4><a name="HLLP_CARDINALITY"></a><tt>HLLP_CARDINALITY</tt></h4> + +<ul> + +<li>Description: Returns HyperLogLogPlus-estimated cardinality for this set. See <a href="HLLP.html">HLLP README</a></li> + +<li>Input: + +<ul> + +<li>hyperLogLogPlus - the hllp set</li> + </ul></li> + +<li>Returns: Long value representing the cardinality for this set. Cardinality of a null set is 0.</li> +</ul></div> +<div class="section"> +<h4><a name="HLLP_INIT"></a><tt>HLLP_INIT</tt></h4> + +<ul> + +<li>Description: Initializes the HyperLogLogPlus estimator set. p must be a value between 4 and sp and sp must be less than 32 and greater than 4. See <a href="HLLP.html">HLLP README</a></li> + +<li>Input: + +<ul> + +<li>p - the precision value for the normal set</li> + +<li>sp - the precision value for the sparse set. If p is set, but sp is 0 or not specified, the sparse set will be disabled.</li> + </ul></li> + +<li>Returns: A new HyperLogLogPlus set</li> +</ul></div> +<div class="section"> +<h4><a name="HLLP_MERGE"></a><tt>HLLP_MERGE</tt></h4> + +<ul> + +<li>Description: Merge hllp sets together. The resulting estimator is initialized with p and sp precision values from the first provided hllp estimator set. See <a href="HLLP.html">HLLP README</a></li> + +<li>Input: + +<ul> + +<li>hllp - List of hllp estimators to merge. Takes a single hllp set or a list.</li> + </ul></li> + +<li>Returns: A new merged HyperLogLogPlus estimator set. Passing an empty list returns null.</li> +</ul></div></div> +<div class="section"> +<h3><a name="Mathematical_Functions"></a>Mathematical Functions</h3> +<div class="section"> +<h4><a name="ABS"></a><tt>ABS</tt></h4> + +<ul> + +<li>Description: Returns the absolute value of a number.</li> + +<li>Input: + +<ul> + +<li>number - The number to take the absolute value of</li> + </ul></li> + +<li>Returns: The absolute value of the number passed in.</li> +</ul></div> +<div class="section"> +<h4><a name="BIN"></a><tt>BIN</tt></h4> + +<ul> + +<li>Description: Computes the bin that the value is in given a set of bounds.</li> + +<li>Input: + +<ul> + +<li>value - The value to bin</li> + +<li>bounds - A list of value bounds (excluding min and max) in sorted order.</li> + </ul></li> + +<li>Returns: Which bin N the value falls in such that bound(N-1) < value <= bound(N). No min and max bounds are provided, so values smaller than the 0’th bound go in the 0’th bin, and values greater than the last bound go in the M’th bin.</li> +</ul></div></div> +<div class="section"> +<h3><a name="Distributional_Statistics"></a>Distributional Statistics</h3> +<div class="section"> +<h4><a name="STATS_ADD"></a><tt>STATS_ADD</tt></h4> + +<ul> + +<li>Description: Adds one or more input values to those that are used to calculate the summary statistics.</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object. If null, then a new one is initialized.</li> + +<li>value+ - One or more numbers to add</li> + </ul></li> + +<li>Returns: A Stellar statistics object</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_BIN"></a><tt>STATS_BIN</tt></h4> + +<ul> + +<li>Description: Computes the bin that the value is in based on the statistical distribution.</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + +<li>value - The value to bin</li> + +<li>bounds? - A list of percentile bin bounds (excluding min and max) or a string representing a known and common set of bins. For convenience, we have provided QUARTILE, QUINTILE, and DECILE which you can pass in as a string arg. If this argument is omitted, then we assume a Quartile bin split.</li> + </ul></li> + +<li>Returns: "Which bin N the value falls in such that bound(N-1) < value <= bound(N). No min and max bounds are provided, so values smaller than the 0’th bound go in the 0’th bin, and values greater than the last bound go in the M’th bin.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_COUNT"></a><tt>STATS_COUNT</tt></h4> + +<ul> + +<li>Description: Calculates the count of the values accumulated (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The count of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_GEOMETRIC_MEAN"></a><tt>STATS_GEOMETRIC_MEAN</tt></h4> + +<ul> + +<li>Description: Calculates the geometric mean of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The geometric mean of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_INIT"></a><tt>STATS_INIT</tt></h4> + +<ul> + +<li>Description: Initializes a statistics object</li> + +<li>Input: + +<ul> + +<li>window_size - The number of input data values to maintain in a rolling window in memory. If window_size is equal to 0, then no rolling window is maintained. Using no rolling window is less memory intensive, but cannot calculate certain statistics like percentiles and kurtosis.</li> + </ul></li> + +<li>Returns: A Stellar statistics object</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_KURTOSIS"></a><tt>STATS_KURTOSIS</tt></h4> + +<ul> + +<li>Description: Calculates the kurtosis of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The kurtosis of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_MAX"></a><tt>STATS_MAX</tt></h4> + +<ul> + +<li>Description: Calculates the maximum of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The maximum of the accumulated values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_MEAN"></a><tt>STATS_MEAN</tt></h4> + +<ul> + +<li>Description: Calculates the mean of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The mean of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_MERGE"></a><tt>STATS_MERGE</tt></h4> + +<ul> + +<li>Description: Merges statistics objects.</li> + +<li>Input: + +<ul> + +<li>statistics - A list of statistics objects</li> + </ul></li> + +<li>Returns: A Stellar statistics object</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_MIN"></a><tt>STATS_MIN</tt></h4> + +<ul> + +<li>Description: Calculates the minimum of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The minimum of the accumulated values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_PERCENTILE"></a><tt>STATS_PERCENTILE</tt></h4> + +<ul> + +<li>Description: Computes the p’th percentile of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + +<li>p - a double where 0 <= p < 1 representing the percentile</li> + </ul></li> + +<li>Returns: The p’th percentile of the data or NaN if the statistics object is null</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_POPULATION_VARIANCE"></a><tt>STATS_POPULATION_VARIANCE</tt></h4> + +<ul> + +<li>Description: Calculates the population variance of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The population variance of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_QUADRATIC_MEAN"></a><tt>STATS_QUADRATIC_MEAN</tt></h4> + +<ul> + +<li>Description: Calculates the quadratic mean of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The quadratic mean of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_SD"></a><tt>STATS_SD</tt></h4> + +<ul> + +<li>Description: Calculates the standard deviation of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The standard deviation of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_SKEWNESS"></a><tt>STATS_SKEWNESS</tt></h4> + +<ul> + +<li>Description: Calculates the skewness of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The skewness of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_SUM"></a><tt>STATS_SUM</tt></h4> + +<ul> + +<li>Description: Calculates the sum of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The sum of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_SUM_LOGS"></a><tt>STATS_SUM_LOGS</tt></h4> + +<ul> + +<li>Description: Calculates the sum of the (natural) log of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The sum of the (natural) log of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_SUM_SQUARES"></a><tt>STATS_SUM_SQUARES</tt></h4> + +<ul> + +<li>Description: Calculates the sum of the squares of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The sum of the squares of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_VARIANCE"></a><tt>STATS_VARIANCE</tt></h4> + +<ul> + +<li>Description: Calculates the variance of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The variance of the values in the window or NaN if the statistics object is null.</li> +</ul></div></div> +<div class="section"> +<h3><a name="Statistical_Outlier_Detection"></a>Statistical Outlier Detection</h3> +<div class="section"> +<h4><a name="OUTLIER_MAD_STATE_MERGE"></a><tt>OUTLIER_MAD_STATE_MERGE</tt></h4> + +<ul> + +<li>Description: Update the statistical state required to compute the Median Absolute Deviation.</li> + +<li>Input: + +<ul> + +<li>[state] - A list of Median Absolute Deviation States to merge. Generally these are states across time.</li> + +<li>currentState? - The current state (optional)</li> + </ul></li> + +<li>Returns: The Median Absolute Deviation state</li> +</ul></div> +<div class="section"> +<h4><a name="OUTLIER_MAD_ADD"></a><tt>OUTLIER_MAD_ADD</tt></h4> + +<ul> + +<li>Description: Add a piece of data to the state.</li> + +<li>Input: + +<ul> + +<li>state - The MAD state</li> + +<li>value - The numeric value to add</li> + </ul></li> + +<li>Returns: The MAD state</li> +</ul></div> +<div class="section"> +<h4><a name="OUTLIER_MAD_SCORE"></a><tt>OUTLIER_MAD_SCORE</tt></h4> + +<ul> + +<li>Description: Get the modified z-score normalized by the MAD: scale * | x_i - median(X) | / MAD. See the first page of <a class="externalLink" href="http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf";>http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf</a></li> + +<li>Input: + +<ul> + +<li>state - The MAD state</li> + +<li>value - The numeric value to score</li> + +<li>scale? - Optionally the scale to use when computing the modified z-score. Default is <tt>0.6745</tt>, see the first page of <a class="externalLink" href="http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf";>http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf</a></li> + </ul></li> + +<li>Returns: The modified z-score</li> +</ul> +<p><a name="Outlier_Analysis"></a></p> +<h1>Outlier Analysis</h1> +<p>A common desire is to find anomalies in numerical data. To that end, we have some simple statistical anomaly detectors.</p></div></div></div> +<div class="section"> +<h2><a name="Median_Absolute_Deviation"></a>Median Absolute Deviation</h2> +<p>Much has been written about this robust estimator. See the first page of <a class="externalLink" href="http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf";>http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf</a> for a good coverage of the good and the bad of MAD. The usage, however is fairly straightforward:</p> + +<ul> + +<li>Gather the statistical state required to compute the MAD + +<ul> + +<li>The distribution of the values of a univariate random variable over time.</li> + +<li>The distribution of the absolute deviations of the values from the median.</li> + </ul></li> + +<li>Use this statistical state to score unseen values. The higher the score, the more unlike the previously seen data the value is.</li> +</ul> +<p>There are a couple of issues which make MAD a bit hard to compute. First, the statistical state requires computing median, which can be computationally expensive to compute exactly. To get around this, we use the OnlineStatisticalProvider to compute a sketch rather than the exact median. Secondly, the statistical state for seasonal data should be limited to a fixed, trailing window. We do this by ensuring that the MAD state is mergeable and able to be queried from within the Profiler.</p> +<div class="section"> +<h3><a name="Example"></a>Example</h3> +<p>We will create a dummy data stream of gaussian noise to illustrate how to use the MAD functionality along with the profiler to tag messages as outliers or not.</p> +<p>To do this, we will create a </p> + +<ul> + +<li>data generator</li> + +<li>parser</li> + +<li>profiler profile</li> + +<li>enrichment and threat triage</li> +</ul> +<div class="section"> +<h4><a name="Data_Generator"></a>Data Generator</h4> +<p>We can create a simple python script to generate a stream of gaussian noise at the frequency of one message per second as a python script which should be saved at <tt>~/rand_gen.py</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>#!/usr/bin/python +import random +import sys +import time +def main(): + mu = float(sys.argv[1]) + sigma = float(sys.argv[2]) + freq_s = int(sys.argv[3]) + while True: + print str(random.gauss(mu, sigma)) + sys.stdout.flush() + time.sleep(freq_s) + +if __name__ == '__main__': + main() +</pre></div></div> +<p>This script will take the following as arguments:</p> + +<ul> + +<li>The mean of the data generated</li> + +<li>The standard deviation of the data generated</li> + +<li>The frequency (in seconds) of the data generated</li> +</ul> +<p>If, however, you’d like to test a longer tailed distribution, like the student t-distribution and have numpy installed, you can use the following as <tt>~/rand_gen.py</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>#!/usr/bin/python +import random +import sys +import time +import numpy as np + +def main(): + df = float(sys.argv[1]) + freq_s = int(sys.argv[2]) + while True: + print str(np.random.standard_t(df)) + sys.stdout.flush() + time.sleep(freq_s) + +if __name__ == '__main__': + main() +</pre></div></div> +<p>This script will take the following as arguments:</p> + +<ul> + +<li>The degrees of freedom for the distribution</li> + +<li>The frequency (in seconds) of the data generated</li> +</ul></div> +<div class="section"> +<h4><a name="The_Parser"></a>The Parser</h4> +<p>We will create a parser that will take the single numbers in and create a message with a field called <tt>value</tt> in them using the <tt>CSVParser</tt>.</p> +<p>Add the following file to <tt>$METRON_HOME/config/zookeeper/parsers/mad.json</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "parserClassName" : "org.apache.metron.parsers.csv.CSVParser" + ,"sensorTopic" : "mad" + ,"parserConfig" : { + "columns" : { + "value_str" : 0 + } + } + ,"fieldTransformations" : [ + { + "transformation" : "STELLAR" + ,"output" : [ "value" ] + ,"config" : { + "value" : "TO_DOUBLE(value_str)" + } + } + ] +} +</pre></div></div></div> +<div class="section"> +<h4><a name="Enrichment_and_Threat_Intel"></a>Enrichment and Threat Intel</h4> +<p>We will set a threat triage level of <tt>10</tt> if a message generates a outlier score of more than 3.5. This cutoff will depend on your data and should be adjusted based on the assumed underlying distribution. Note that under the assumptions of normality, MAD will act as a robust estimator of the standard deviation, so the cutoff should be considered the number of standard deviations away. For other distributions, there are other interpretations which will make sense in the context of measuring the “degree different”. See <a class="externalLink" href="http://eurekastatistics.com/using-the-median-absolute-deviation-to-find-outliers/";>http://eurekastatistics.com/using-the-median-absolute-deviation-to-find-outliers/</a> for a brief discussion of this.</p> +<p>Create the following in <tt>$METRON_HOME/config/zookeeper/enrichments/mad.json</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "enrichment": { + "fieldMap": { + "stellar" : { + "config" : { + "parser_score" : "OUTLIER_MAD_SCORE(OUTLIER_MAD_STATE_MERGE( +PROFILE_GET( 'sketchy_mad', 'global', PROFILE_FIXED(10, 'MINUTES')) ), value)" + ,"is_alert" : "if parser_score > 3.5 then true else is_alert" + } + } + } + ,"fieldToTypeMap": { } + }, + "threatIntel": { + "fieldMap": { }, + "fieldToTypeMap": { }, + "triageConfig" : { + "riskLevelRules" : [ + { + "rule" : "parser_score > 3.5", + "score" : 10 + } + ], + "aggregator" : "MAX" + } + } +} +</pre></div></div> +<p>We also need an indexing configuration. Create the following in <tt>$METRON_HOME/config/zookeeper/indexing/mad.json</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "hdfs" : { + "index": "mad", + "batchSize": 1, + "enabled" : true + }, + "elasticsearch" : { + "index": "mad", + "batchSize": 1, + "enabled" : true + } +} +</pre></div></div></div> +<div class="section"> +<h4><a name="The_Profiler"></a>The Profiler</h4> +<p>We can set up the profiler to track the MAD statistical state required to compute MAD. For the purposes of this demonstration, we will configure the profiler to capture statistics on the minute mark. We will capture a global statistical state for the <tt>value</tt> field and we will look back for a 5 minute window when computing the median.</p> +<p>Create the following file at <tt>$METRON_HOME/config/zookeeper/profiler.json</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "profiles": [ + { + "profile": "sketchy_mad", + "foreach": "'global'", + "onlyif": "true", + "init" : { + "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', +'global', PROFILE_FIXED(5, 'MINUTES')))" + }, + "update": { + "s": "OUTLIER_MAD_ADD(s, value)" + }, + "result": "s" + } + ] +} +</pre></div></div> +<p>Adjust <tt>$METRON_HOME/config/zookeeper/global.json</tt> to adjust the capture duration:</p> + +<div class="source"> +<div class="source"> +<pre> "profiler.client.period.duration" : "1", + "profiler.client.period.duration.units" : "MINUTES" +</pre></div></div> +<p>Adjust <tt>$METRON_HOME/config/profiler.properties</tt> to adjust the capture duration by changing <tt>profiler.period.duration=15</tt> to <tt>profiler.period.duration=1</tt></p></div> +<div class="section"> +<h4><a name="Execute_the_Flow"></a>Execute the Flow</h4> + +<ol style="list-style-type: decimal"> + +<li> +<p>Install the elasticsearch head plugin by executing: <tt>/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head</tt></p></li> + +<li> +<p>Stopping all other parser topologies via monit</p></li> + +<li> +<p>Create the <tt>mad</tt> kafka topic by executing: <tt>/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic mad --partitions 1 --replication-factor 1</tt></p></li> + +<li> +<p>Push the modified configs by executing: <tt>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z node1:2181 -i $METRON_HOME/config/zookeeper/</tt></p></li> + +<li> +<p>Start the profiler by executing: <tt>$METRON_HOME/bin/start_profiler_topology.sh</tt></p></li> + +<li> +<p>Start the parser topology by executing: <tt>$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s mad</tt></p></li> + +<li> +<p>Ensure that the enrichment and indexing topologies are started. If not, then start those via monit or by hand.</p></li> + +<li> +<p>Generate data into kafka by executing the following for at least 10 minutes: <tt>~/rand_gen.py 0 1 1 | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic mad</tt> Note: if you chose the use the t-distribution script above, you would adjust the parameters of the <tt>rand_gen.py</tt> script accordingly.</p></li> + +<li> +<p>Stop the above with ctrl-c and send in an obvious outlier into kafka: <tt>echo "1000" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic mad</tt></p></li> +</ol> +<p>You should be able to find the outlier via the elasticsearch head plugin by searching for the messages where <tt>is_alert</tt> is <tt>true</tt>.</p></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
Added: release/metron/0.4.0/site-book/metron-deployment/Kerberos-ambari-setup.html ============================================================================== --- release/metron/0.4.0/site-book/metron-deployment/Kerberos-ambari-setup.html (added) +++ release/metron/0.4.0/site-book/metron-deployment/Kerberos-ambari-setup.html Wed Jul 5 06:56:42 2017 @@ -0,0 +1,275 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-06-27 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Setting Up Kerberos in Vagrant Full Dev</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Setting Up Kerberos in Vagrant Full Dev</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-down"></i> + Deployment</a> + <ul class="nav nav-list"> + + <li class="active"> + + <a href="#"><i class="none"></i>Kerberos-ambari-setup</a> + </li> + + <li> + + <a href="../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"> + <i class="none"></i> + Kerberos-manual-setup</a> + </li> + + <li> + + <a href="../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"> + <i class="none"></i> + Amazon-ec2</a> + </li> + + <li> + + <a href="../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"> + <i class="none"></i> + Ansible-docker</a> + </li> + + <li> + + <a href="../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"> + <i class="none"></i> + Rpm-docker</a> + </li> + + <li> + + <a href="../metron-deployment/packaging/packer-build/index.html" title="Packer-build"> + <i class="none"></i> + Packer-build</a> + </li> + + <li> + + <a href="../metron-deployment/roles/index.html" title="Roles"> + <i class="icon-chevron-right"></i> + Roles</a> + </li> + + <li> + + <a href="../metron-deployment/vagrant/index.html" title="Vagrant"> + <i class="icon-chevron-right"></i> + Vagrant</a> + </li> + </ul> + </li> + + <li> + + <a href="../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Setting Up Kerberos in Vagrant Full Dev</h1> +<p><a name="Setting_Up_Kerberos_in_Vagrant_Full_Dev"></a></p> +<p><b>Note:</b> These are instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the sensor connections or MAAS. General Kerberization notes can be found in the metron-deployment <a href="../index.html">README.md</a></p> +<div class="section"> +<h2><a name="Setup_a_KDC"></a>Setup a KDC</h2> +<p>See <a href="Kerberos-manual-setup.html#Setup_a_KDC">Setup a KDC</a> and <a href="Kerberos-manual-setup.html#Verify_KDC">Verify KDC</a></p></div> +<div class="section"> +<h2><a name="Ambari_Setup"></a>Ambari Setup</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Kerberize the cluster via Ambari. More detailed documentation can be found <a class="externalLink" href="http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html";>here</a>.</p> +<p>a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)</p> +<p><img src="../images/enable-kerberos.png" alt="enable keberos" /></p> +<p><img src="../images/enable-kerberos-started.png" alt="enable keberos get started" /></p> +<p>b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/ad...@example.com when testing the KDC. Use the password you entered during the step for adding the admin principal.</p> +<p><img src="../images/enable-kerberos-configure-kerberos.png" alt="enable keberos configure" /></p> +<p>c. Click through to “Start and Test Services.” Let the cluster spin up.</p></li> +</ol></div> +<div class="section"> +<h2><a name="Push_Data"></a>Push Data</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Kinit with the metron user</p> + +<div class="source"> +<div class="source"> +<pre>kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com +</pre></div></div></li> +</ol> +<p>See <a href="Kerberos-manual-setup.html#Push_Data">Push Data</a></p> +<div class="section"> +<h3><a name="More_Information"></a>More Information</h3> +<p>See <a href="Kerberos-manual-setup.html#More_Information">More Information</a></p></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Added: release/metron/0.4.0/site-book/metron-deployment/Kerberos-manual-setup.html ============================================================================== --- release/metron/0.4.0/site-book/metron-deployment/Kerberos-manual-setup.html (added) +++ release/metron/0.4.0/site-book/metron-deployment/Kerberos-manual-setup.html Wed Jul 5 06:56:42 2017 @@ -0,0 +1,803 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-06-27 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Kerberos Setup</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Kerberos Setup</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-down"></i> + Deployment</a> + <ul class="nav nav-list"> + + <li> + + <a href="../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"> + <i class="none"></i> + Kerberos-ambari-setup</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Kerberos-manual-setup</a> + </li> + + <li> + + <a href="../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"> + <i class="none"></i> + Amazon-ec2</a> + </li> + + <li> + + <a href="../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"> + <i class="none"></i> + Ansible-docker</a> + </li> + + <li> + + <a href="../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"> + <i class="none"></i> + Rpm-docker</a> + </li> + + <li> + + <a href="../metron-deployment/packaging/packer-build/index.html" title="Packer-build"> + <i class="none"></i> + Packer-build</a> + </li> + + <li> + + <a href="../metron-deployment/roles/index.html" title="Roles"> + <i class="icon-chevron-right"></i> + Roles</a> + </li> + + <li> + + <a href="../metron-deployment/vagrant/index.html" title="Vagrant"> + <i class="icon-chevron-right"></i> + Vagrant</a> + </li> + </ul> + </li> + + <li> + + <a href="../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Kerberos Setup</h1> +<p>This document provides instructions for kerberizing Metron’s Vagrant-based development environments; “Quick Dev” and “Full Dev”. These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment <a href="../index.html">README.md</a>.</p> + +<ul> + +<li><a href="#Setup">Setup</a></li> + +<li><a href="#Setup_a_KDC">Setup a KDC</a></li> + +<li><a href="#Verify_KDC">Verify KDC</a></li> + +<li><a href="#Enable_Kerberos">Enable Kerberos</a></li> + +<li><a href="#Kafka_Authorization">Kafka Authorization</a></li> + +<li><a href="#HBase_Authorization">HBase Authorization</a></li> + +<li><a href="#Storm_Authorization">Storm Authorization</a></li> + +<li><a href="#Start_Metron">Start Metron</a></li> + +<li><a href="#Push_Data">Push Data</a></li> + +<li><a href="#More_Information">More Information</a></li> +</ul> +<div class="section"> +<h2><a name="Setup"></a>Setup</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Deploy a Vagrant development environment; either <a href="vagrant/full-dev-platform/index.html">Full Dev</a> or <a href="vagrant/quick-dev-platform/index.html">Quick Dev</a>.</p></li> + +<li> +<p>Export the following environment variables. These need to be set for the remainder of the instructions. Replace <tt>node1</tt> with the appropriate hosts, if you are running Metron anywhere other than Vagrant.</p> + +<div class="source"> +<div class="source"> +<pre># execute as root +sudo su - +export KAFKA_HOME="/usr/hdp/current/kafka-broker" +export ZOOKEEPER=node1:2181 +export ELASTICSEARCH=node1:9200 +export BROKERLIST=node1:6667 +export HDP_HOME="/usr/hdp/current" +export KAFKA_HOME="${HDP_HOME}/kafka-broker" +export METRON_VERSION="0.4.0" +export METRON_HOME="/usr/metron/${METRON_VERSION}" +</pre></div></div></li> + +<li> +<p>Execute the following commands as root.</p> + +<div class="source"> +<div class="source"> +<pre>sudo su - +</pre></div></div></li> + +<li> +<p>Stop all Metron topologies. They will be restarted again once Kerberos has been enabled.</p> + +<div class="source"> +<div class="source"> +<pre>for topology in bro snort enrichment indexing; do + storm kill $topology; +done +</pre></div></div></li> + +<li> +<p>Create the <tt>metron</tt> user’s home directory in HDFS.</p> + +<div class="source"> +<div class="source"> +<pre>sudo -u hdfs hdfs dfs -mkdir /user/metron +sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron +sudo -u hdfs hdfs dfs -chmod 770 /user/metron +</pre></div></div></li> +</ol></div> +<div class="section"> +<h2><a name="Setup_a_KDC"></a>Setup a KDC</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Install dependencies.</p> + +<div class="source"> +<div class="source"> +<pre>yum -y install krb5-server krb5-libs krb5-workstation +</pre></div></div></li> + +<li> +<p>Define the host, <tt>node1</tt>, as the KDC.</p> + +<div class="source"> +<div class="source"> +<pre>sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf +cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts +</pre></div></div></li> + +<li> +<p>Ensure the KDC can issue renewable tickets. This can be necessary on a real cluster, but should not be on full-dev. In /var/kerberos/krb5kdc/kdc.conf ensure the following is in the realm section</p> + +<div class="source"> +<div class="source"> +<pre>max_renewable_life = 7d +</pre></div></div></li> + +<li> +<p>Do not copy/paste this full set of commands as the <tt>kdb5_util</tt> command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database.</p> + +<div class="source"> +<div class="source"> +<pre>kdb5_util create -s +/etc/rc.d/init.d/krb5kdc start +chkconfig krb5kdc on +/etc/rc.d/init.d/kadmin start +chkconfig kadmin on +</pre></div></div></li> + +<li> +<p>Setup the <tt>admin</tt> and <tt>metron</tt> principals. You’ll <tt>kinit</tt> as the <tt>metron</tt> principal when running topologies. Make sure to remember the passwords.</p> + +<div class="source"> +<div class="source"> +<pre>kadmin.local -q "addprinc admin/admin" +kadmin.local -q "addprinc metron" +</pre></div></div></li> +</ol></div> +<div class="section"> +<h2><a name="Verify_KDC"></a>Verify KDC</h2> +<p>Ticket renewal is by default disallowed in many linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron’s Storm topologies:</p> + +<div class="source"> +<div class="source"> +<pre>Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The TGT found is not renewable +</pre></div></div> +<p>Ensure the Metron keytab is renewable. Look for the ‘R’ flag from the following command</p> + +<div class="source"> +<div class="source"> +<pre>klist -f +</pre></div></div> +<p>If the ‘R’ flags are present, you may skip to next section.</p> +<p>If the ‘R’ flags are absent, you will need to follow the below steps: If the KDC is already setup, then editing max_life and max_renewable_life in <tt>/var/kerberos/krb5kdc/kdc.conf</tt>, and restarting kadmin and krb5kdc services will not change the policies for existing users. </p> +<p>You need to set the renew lifetime for existing users and krbtgt realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:</p> + +<div class="source"> +<div class="source"> +<pre>kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable krbtgt/example....@example.com" +kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable met...@example.com" +</pre></div></div></div> +<div class="section"> +<h2><a name="Enable_Kerberos"></a>Enable Kerberos</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>In <a class="externalLink" href="http://node1:8080";>Ambari</a>, setup Storm to use Kerberos and run worker jobs as the submitting user.</p> +<p>a. Add the following properties to the custom storm-site:</p> + +<div class="source"> +<div class="source"> +<pre>topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT'] +nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT'] +supervisor.run.worker.as.user=true +</pre></div></div> +<p>b. In the Storm config section in Ambari, choose “Add Property” under custom storm-site:</p> +<p><img src="../images/ambari-storm-site.png" alt="custom storm-site" /></p> +<p>c. In the dialog window, choose the “bulk property add mode” toggle button and add the below values:</p> +<p><img src="../images/ambari-storm-site-properties.png" alt="custom storm-site properties" /></p></li> + +<li> +<p>Kerberize the cluster via Ambari. More detailed documentation can be found <a class="externalLink" href="http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html";>here</a>.</p> +<p>a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)</p> +<p><img src="../images/enable-kerberos.png" alt="enable keberos" /></p> +<p><img src="../images/enable-kerberos-started.png" alt="enable keberos get started" /></p> +<p>b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/ad...@example.com when testing the KDC. Use the password you entered during the step for adding the admin principal.</p> +<p><img src="../images/enable-kerberos-configure-kerberos.png" alt="enable keberos configure" /></p> +<p>c. Click through to “Start and Test Services.” Let the cluster spin up, but don’t worry about starting up Metron via Ambari - we’re going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click “continue.” When you’re finished, the custom storm-site should look similar to the following:</p> +<p><img src="../images/custom-storm-site-final.png" alt="enable keberos configure" /></p></li> + +<li> +<p>Create a Metron keytab</p> + +<div class="source"> +<div class="source"> +<pre>kadmin.local -q "ktadd -k metron.headless.keytab met...@example.com" +cp metron.headless.keytab /etc/security/keytabs +chown metron:hadoop /etc/security/keytabs/metron.headless.keytab +chmod 440 /etc/security/keytabs/metron.headless.keytab +</pre></div></div></li> +</ol></div> +<div class="section"> +<h2><a name="Kafka_Authorization"></a>Kafka Authorization</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Acquire a Kerberos ticket using the <tt>metron</tt> principal.</p> + +<div class="source"> +<div class="source"> +<pre>kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com +</pre></div></div></li> + +<li> +<p>Create any additional Kafka topics that you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. For example, you may want to add a topic for ‘yaf’ telemetry.</p> + +<div class="source"> +<div class="source"> +<pre>${KAFKA_HOME}/bin/kafka-topics.sh \ + --zookeeper ${ZOOKEEPER} \ + --create \ + --topic yaf \ + --partitions 1 \ + --replication-factor 1 +</pre></div></div></li> + +<li> +<p>Setup Kafka ACLs for the <tt>bro</tt>, <tt>snort</tt>, <tt>enrichments</tt>, and <tt>indexing</tt> topics. Run the same command against any additional topics that you might be using; for example <tt>yaf</tt>.</p> + +<div class="source"> +<div class="source"> +<pre>export KERB_USER=metron +for topic in bro snort enrichments indexing; do + ${KAFKA_HOME}/bin/kafka-acls.sh \ + --authorizer kafka.security.auth.SimpleAclAuthorizer \ + --authorizer-properties zookeeper.connect=${ZOOKEEPER} \ + --add \ + --allow-principal User:${KERB_USER} \ + --topic ${topic} +done +</pre></div></div></li> + +<li> +<p>Setup Kafka ACLs for the consumer groups. This command sets the ACLs for Bro, Snort, YAF, Enrichments, Indexing, and the Profiler. Execute the same command for any additional Parsers that you may be running.</p> + +<div class="source"> +<div class="source"> +<pre>export KERB_USER=metron +for group in bro_parser snort_parser yaf_parser enrichments indexing profiler; do + ${KAFKA_HOME}/bin/kafka-acls.sh \ + --authorizer kafka.security.auth.SimpleAclAuthorizer \ + --authorizer-properties zookeeper.connect=${ZOOKEEPER} \ + --add \ + --allow-principal User:${KERB_USER} \ + --group ${group} +done +</pre></div></div></li> + +<li> +<p>Add the <tt>metron</tt> principal to the <tt>kafka-cluster</tt> ACL.</p> + +<div class="source"> +<div class="source"> +<pre>${KAFKA_HOME}/bin/kafka-acls.sh \ + --authorizer kafka.security.auth.SimpleAclAuthorizer \ + --authorizer-properties zookeeper.connect=${ZOOKEEPER} \ + --add \ + --allow-principal User:${KERB_USER} \ + --cluster kafka-cluster +</pre></div></div></li> +</ol></div> +<div class="section"> +<h2><a name="HBase_Authorization"></a>HBase Authorization</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Acquire a Kerberos ticket using the <tt>hbase</tt> principal</p> + +<div class="source"> +<div class="source"> +<pre>kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_clus...@example.com +</pre></div></div></li> + +<li> +<p>Grant permissions for the HBase tables used in Metron.</p> + +<div class="source"> +<div class="source"> +<pre>echo "grant 'metron', 'RW', 'threatintel'" | hbase shell +echo "grant 'metron', 'RW', 'enrichment'" | hbase shell +</pre></div></div></li> + +<li> +<p>If you are using the Profiler, do the same for its HBase table.</p> + +<div class="source"> +<div class="source"> +<pre>echo "create 'profiler', 'P'" | hbase shell +echo "grant 'metron', 'RW', 'profiler', 'P'" | hbase shell +</pre></div></div></li> +</ol></div> +<div class="section"> +<h2><a name="Storm_Authorization"></a>Storm Authorization</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Switch to the <tt>metron</tt> user and acquire a Kerberos ticket for the <tt>metron</tt> principal.</p> + +<div class="source"> +<div class="source"> +<pre>su metron +kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com +</pre></div></div></li> + +<li> +<p>Create the directory <tt>/home/metron/.storm</tt> and switch to that directory.</p> + +<div class="source"> +<div class="source"> +<pre>mkdir /home/metron/.storm +cd /home/metron/.storm +</pre></div></div></li> + +<li> +<p>Ensure the Metron keytab is renewable. See <a href="#Verify_KDC">Verify KDC</a> above.</p></li> + +<li> +<p>Create a client JAAS file at <tt>/home/metron/.storm/client_jaas.conf</tt>. This should look identical to the Storm client JAAS file located at <tt>/etc/storm/conf/client_jaas.conf</tt> except for the addition of a <tt>Client</tt> stanza. The <tt>Client</tt> stanza is used for Zookeeper. All quotes and semicolons are necessary.</p> + +<div class="source"> +<div class="source"> +<pre>cat << EOF > client_jaas.conf +StormClient { + com.sun.security.auth.module.Krb5LoginModule required + useTicketCache=true + renewTicket=true + serviceName="nimbus"; +}; +Client { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + keyTab="/etc/security/keytabs/metron.headless.keytab" + storeKey=true + useTicketCache=false + serviceName="zookeeper" + principal="met...@example.com"; +}; +KafkaClient { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + keyTab="/etc/security/keytabs/metron.headless.keytab" + storeKey=true + useTicketCache=false + serviceName="kafka" + principal="met...@example.com"; +}; +EOF +</pre></div></div></li> + +<li> +<p>Create a YAML file at <tt>/home/metron/.storm/storm.yaml</tt>. This should point to the client JAAS file. Set the array of nimbus hosts accordingly.</p> + +<div class="source"> +<div class="source"> +<pre>cat << EOF > /home/metron/.storm/storm.yaml +nimbus.seeds : ['node1'] +java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf' +storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin' +EOF +</pre></div></div></li> + +<li> +<p>Create an auxiliary storm configuration file at <tt>/home/metron/storm-config.json</tt>. Note the login config option in the file points to the client JAAS file.</p> + +<div class="source"> +<div class="source"> +<pre>cat << EOF > /home/metron/storm-config.json +{ + "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf" +} +EOF +</pre></div></div></li> + +<li> +<p>Configure the Enrichment, Indexing and Profiler topologies to use the client JAAS file. To do this, the following key-value pairs:</p> + +<ul> + +<li><tt>kafka.security.protocol=PLAINTEXTSASL</tt></li> + +<li><tt>topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf</tt></li> + </ul> +<p>must be added to each of the topology properties files:</p> + +<ul> + +<li><tt>${METRON_HOME}/config/enrichment.properties</tt></li> + +<li><tt>${METRON_HOME}/config/elasticsearch.properties</tt></li> + +<li><tt>${METRON_HOME}/config/profiler.properties</tt></li> + </ul> +<p>You may use the following command to automate this step:</p> + +<div class="source"> +<div class="source"> +<pre>for file in enrichment.properties elasticsearch.properties profiler.properties; do + echo ${file} + sed -i "s/^kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/" "${METRON_HOME}/config/${file}" + sed -i "s/^topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/" "${METRON_HOME}/config/${file}" +done +</pre></div></div></li> +</ol></div> +<div class="section"> +<h2><a name="Start_Metron"></a>Start Metron</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Switch to the <tt>metron</tt> user and acquire a Kerberos ticket for the <tt>metron</tt> principal.</p> + +<div class="source"> +<div class="source"> +<pre>su metron +kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com +</pre></div></div></li> + +<li> +<p>Restart the parser topologies. Be sure to pass in the new parameter, <tt>-ksp</tt> or <tt>--kafka_security_protocol</tt>. The following command will start only the Bro and Snort topologies. Execute the same command for any other Parsers that you may need, for example <tt>yaf</tt>.</p> + +<div class="source"> +<div class="source"> +<pre>for parser in bro snort; do + ${METRON_HOME}/bin/start_parser_topology.sh \ + -z ${ZOOKEEPER} \ + -s ${parser} \ + -ksp SASL_PLAINTEXT \ + -e /home/metron/storm-config.json; +done +</pre></div></div></li> + +<li> +<p>Restart the Enrichment and Indexing topologies.</p> + +<div class="source"> +<div class="source"> +<pre>${METRON_HOME}/bin/start_enrichment_topology.sh +${METRON_HOME}/bin/start_elasticsearch_topology.sh +</pre></div></div></li> +</ol> +<p>Metron should be ready to receive data.</p></div> +<div class="section"> +<h2><a name="Push_Data"></a>Push Data</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Push some sample data to one of the parser topics. E.g for Bro we took raw data from <a href="../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput/index.html">metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput</a></p> + +<div class="source"> +<div class="source"> +<pre>cat sample-bro.txt | ${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \ + --broker-list ${BROKERLIST} \ + --security-protocol SASL_PLAINTEXT \ + --topic bro +</pre></div></div></li> + +<li> +<p>Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indices. Replace yaf with whichever parser type you’ve chosen.</p> + +<div class="source"> +<div class="source"> +<pre>curl -XGET "${ELASTICSEARCH}/bro*/_search" +curl -XGET "${ELASTICSEARCH}/bro*/_count" +</pre></div></div></li> + +<li> +<p>You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions</p></li> +</ol></div> +<div class="section"> +<h2><a name="More_Information"></a>More Information</h2> +<div class="section"> +<h3><a name="Kerberos"></a>Kerberos</h3> +<p>Unsure of your Kerberos principal associated with a keytab? There are a couple ways to get this. One is via the list of principals that Ambari provides via downloadable csv. If you didn’t download this list, you can also check the principal manually by running the following against the keytab.</p> + +<div class="source"> +<div class="source"> +<pre>klist -kt /etc/security/keytabs/<keytab-file-name> +</pre></div></div> +<p>E.g.</p> + +<div class="source"> +<div class="source"> +<pre>klist -kt /etc/security/keytabs/hbase.headless.keytab +Keytab name: FILE:/etc/security/keytabs/hbase.headless.keytab +KVNO Timestamp Principal +---- ----------------- -------------------------------------------------------- + 1 03/28/17 19:29:36 hbase-metron_clus...@example.com + 1 03/28/17 19:29:36 hbase-metron_clus...@example.com + 1 03/28/17 19:29:36 hbase-metron_clus...@example.com + 1 03/28/17 19:29:36 hbase-metron_clus...@example.com + 1 03/28/17 19:29:36 hbase-metron_clus...@example.com +</pre></div></div></div> +<div class="section"> +<h3><a name="Kafka_with_Kerberos_enabled"></a>Kafka with Kerberos enabled</h3> +<div class="section"> +<h4><a name="Running_Sensors"></a>Running Sensors</h4> +<p>A couple steps are required to produce data to a Kerberized Kafka topic. On the host you’ll be setting up your sensor(s), switch to the metron user and create a client_jaas.conf file in the metron home directory if one doesn’t already exist. It should be owned by metron:metron and contain at least the following stanza that tells the Kafka client how to interact with Kerberos:</p> + +<div class="source"> +<div class="source"> +<pre>su - metron +cat ${METRON_HOME}/client_jaas.conf +... +KafkaClient { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + keyTab="/etc/security/keytabs/metron.headless.keytab" + storeKey=true + useTicketCache=false + serviceName="kafka" + principal="met...@example.com"; +}; +</pre></div></div> +<p>You’ll also need to set KAFKA_OPTS to tell the Kafka client how to interact with Kerberos.</p> + +<div class="source"> +<div class="source"> +<pre>export KAFKA_OPTS="-Djava.security.auth.login.config=${METRON_HOME}/client_jaas.conf" +</pre></div></div> +<p>For sensors that leverage the Kafka console producer to pipe data into Metron, e.g. Snort and Yaf, you will need to modify the corresponding sensor shell scripts or config to append the SASL security protocol property. <tt>--security-protocol SASL_PLAINTEXT</tt>. Be sure to kinit with the metron user’s keytab before executing the script that starts the sensor.</p> +<p>More notes can be found in <a href="../metron-sensors/index.html">metron/metron-sensors/README.md</a></p></div> +<div class="section"> +<h4><a name="Write_data_to_a_topic_with_SASL"></a>Write data to a topic with SASL</h4> + +<div class="source"> +<div class="source"> +<pre>cat sample-yaf.txt | ${KAFKA_HOME}/bin/kafka-console-producer.sh \ + --broker-list ${BROKERLIST} \ + --security-protocol PLAINTEXTSASL \ + --topic yaf +</pre></div></div></div> +<div class="section"> +<h4><a name="View_topic_data_from_latest_offset_with_SASL"></a>View topic data from latest offset with SASL</h4> + +<div class="source"> +<div class="source"> +<pre>${KAFKA_HOME}/bin/kafka-console-consumer.sh \ + --zookeeper ${ZOOKEEPER} \ + --security-protocol PLAINTEXTSASL \ + --topic yaf +</pre></div></div></div> +<div class="section"> +<h4><a name="Modify_the_sensor-stubs_to_send_logs_via_SASL"></a>Modify the sensor-stubs to send logs via SASL</h4> + +<div class="source"> +<div class="source"> +<pre>sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub +for sensorstub in bro snort; do + service sensor-stubs stop ${sensorstub}; + service sensor-stubs start ${sensorstub}; +done +</pre></div></div></div></div> +<div class="section"> +<h3><a name="References"></a>References</h3> + +<ul> + +<li><a class="externalLink" href="https://github.com/apache/storm/blob/master/SECURITY.md";>https://github.com/apache/storm/blob/master/SECURITY.md</a></li> +</ul></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
