Added: dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-analytics/metron-statistics/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-analytics/metron-statistics/index.html (added) +++ dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-analytics/metron-statistics/index.html Fri Feb 10 20:21:39 2017 @@ -0,0 +1,916 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-02-10 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170210" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Statistics and Mathematical Functions</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> + Metron-Incubating</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Statistics and Mathematical Functions</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-02-10</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-down"></i> + Analytics</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service"> + <i class="none"></i> + Maas-service</a> + </li> + + <li> + + <a href="../../metron-analytics/metron-profiler/index.html" title="Profiler"> + <i class="none"></i> + Profiler</a> + </li> + + <li> + + <a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client"> + <i class="none"></i> + Profiler-client</a> + </li> + + <li class="active"> + + <a href="#"><i class="icon-chevron-down"></i>Statistics</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-analytics/metron-statistics/HLLP.html" title="HLLP"> + <i class="none"></i> + HLLP</a> + </li> + </ul> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Statistics and Mathematical Functions</h1> +<p><a name="Statistics_and_Mathematical_Functions"></a></p> +<p>A variety of non-trivial and advanced analytics make use of statistics and advanced mathematical functions. Particular, capturing the statistical snapshots in a scalable way can open up doors for more advanced analytics such as outlier analysis. As such, this project is aimed at capturing a robust set of statistical functions and statistical-based algorithms in the form of Stellar functions. These functions can be used from everywhere where Stellar is used.</p> +<div class="section"> +<h2><a name="Stellar_Functions"></a>Stellar Functions</h2> +<div class="section"> +<h3><a name="Approximation_Statistics"></a>Approximation Statistics</h3> +<div class="section"> +<h4><a name="HLLP_ADD"></a><tt>HLLP_ADD</tt></h4> + +<ul> + +<li>Description: Add value to the HyperLogLogPlus estimator set. See <a href="HLLP.html">HLLP README</a></li> + +<li>Input: + +<ul> + +<li>hyperLogLogPlus - the hllp estimator to add a value to</li> + +<li>value+ - value to add to the set. Takes a single item or a list.</li> + </ul></li> + +<li>Returns: The HyperLogLogPlus set with a new value added</li> +</ul></div> +<div class="section"> +<h4><a name="HLLP_CARDINALITY"></a><tt>HLLP_CARDINALITY</tt></h4> + +<ul> + +<li>Description: Returns HyperLogLogPlus-estimated cardinality for this set. See <a href="HLLP.html">HLLP README</a></li> + +<li>Input: + +<ul> + +<li>hyperLogLogPlus - the hllp set</li> + </ul></li> + +<li>Returns: Long value representing the cardinality for this set</li> +</ul></div> +<div class="section"> +<h4><a name="HLLP_INIT"></a><tt>HLLP_INIT</tt></h4> + +<ul> + +<li>Description: Initializes the HyperLogLogPlus estimator set. p must be a value between 4 and sp and sp must be less than 32 and greater than 4. See <a href="HLLP.html">HLLP README</a></li> + +<li>Input: + +<ul> + +<li>p - the precision value for the normal set</li> + +<li>sp - the precision value for the sparse set. If p is set, but sp is 0 or not specified, the sparse set will be disabled.</li> + </ul></li> + +<li>Returns: A new HyperLogLogPlus set</li> +</ul></div> +<div class="section"> +<h4><a name="HLLP_MERGE"></a><tt>HLLP_MERGE</tt></h4> + +<ul> + +<li>Description: Merge hllp sets together. The resulting estimator is initialized with p and sp precision values from the first provided hllp estimator set. See <a href="HLLP.html">HLLP README</a></li> + +<li>Input: + +<ul> + +<li>hllp - List of hllp estimators to merge. Takes a single hllp set or a list.</li> + </ul></li> + +<li>Returns: A new merged HyperLogLogPlus estimator set</li> +</ul></div></div> +<div class="section"> +<h3><a name="Mathematical_Functions"></a>Mathematical Functions</h3> +<div class="section"> +<h4><a name="ABS"></a><tt>ABS</tt></h4> + +<ul> + +<li>Description: Returns the absolute value of a number.</li> + +<li>Input: + +<ul> + +<li>number - The number to take the absolute value of</li> + </ul></li> + +<li>Returns: The absolute value of the number passed in.</li> +</ul></div> +<div class="section"> +<h4><a name="BIN"></a><tt>BIN</tt></h4> + +<ul> + +<li>Description: Computes the bin that the value is in given a set of bounds.</li> + +<li>Input: + +<ul> + +<li>value - The value to bin</li> + +<li>bounds - A list of value bounds (excluding min and max) in sorted order.</li> + </ul></li> + +<li>Returns: Which bin N the value falls in such that bound(N-1) < value <= bound(N). No min and max bounds are provided, so values smaller than the 0’th bound go in the 0’th bin, and values greater than the last bound go in the M’th bin.</li> +</ul></div></div> +<div class="section"> +<h3><a name="Distributional_Statistics"></a>Distributional Statistics</h3> +<div class="section"> +<h4><a name="STATS_ADD"></a><tt>STATS_ADD</tt></h4> + +<ul> + +<li>Description: Adds one or more input values to those that are used to calculate the summary statistics.</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object. If null, then a new one is initialized.</li> + +<li>value+ - One or more numbers to add</li> + </ul></li> + +<li>Returns: A Stellar statistics object</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_BIN"></a><tt>STATS_BIN</tt></h4> + +<ul> + +<li>Description: Computes the bin that the value is in based on the statistical distribution.</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + +<li>value - The value to bin</li> + +<li>bounds? - A list of percentile bin bounds (excluding min and max) or a string representing a known and common set of bins. For convenience, we have provided QUARTILE, QUINTILE, and DECILE which you can pass in as a string arg. If this argument is omitted, then we assume a Quartile bin split.</li> + </ul></li> + +<li>Returns: "Which bin N the value falls in such that bound(N-1) < value <= bound(N). No min and max bounds are provided, so values smaller than the 0’th bound go in the 0’th bin, and values greater than the last bound go in the M’th bin.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_COUNT"></a><tt>STATS_COUNT</tt></h4> + +<ul> + +<li>Description: Calculates the count of the values accumulated (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The count of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_GEOMETRIC_MEAN"></a><tt>STATS_GEOMETRIC_MEAN</tt></h4> + +<ul> + +<li>Description: Calculates the geometric mean of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The geometric mean of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_INIT"></a><tt>STATS_INIT</tt></h4> + +<ul> + +<li>Description: Initializes a statistics object</li> + +<li>Input: + +<ul> + +<li>window_size - The number of input data values to maintain in a rolling window in memory. If window_size is equal to 0, then no rolling window is maintained. Using no rolling window is less memory intensive, but cannot calculate certain statistics like percentiles and kurtosis.</li> + </ul></li> + +<li>Returns: A Stellar statistics object</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_KURTOSIS"></a><tt>STATS_KURTOSIS</tt></h4> + +<ul> + +<li>Description: Calculates the kurtosis of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The kurtosis of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_MAX"></a><tt>STATS_MAX</tt></h4> + +<ul> + +<li>Description: Calculates the maximum of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The maximum of the accumulated values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_MEAN"></a><tt>STATS_MEAN</tt></h4> + +<ul> + +<li>Description: Calculates the mean of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The mean of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_MERGE"></a><tt>STATS_MERGE</tt></h4> + +<ul> + +<li>Description: Merges statistics objects.</li> + +<li>Input: + +<ul> + +<li>statistics - A list of statistics objects</li> + </ul></li> + +<li>Returns: A Stellar statistics object</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_MIN"></a><tt>STATS_MIN</tt></h4> + +<ul> + +<li>Description: Calculates the minimum of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The minimum of the accumulated values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_PERCENTILE"></a><tt>STATS_PERCENTILE</tt></h4> + +<ul> + +<li>Description: Computes the p’th percentile of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + +<li>p - a double where 0 <= p < 1 representing the percentile</li> + </ul></li> + +<li>Returns: The p’th percentile of the data or NaN if the statistics object is null</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_POPULATION_VARIANCE"></a><tt>STATS_POPULATION_VARIANCE</tt></h4> + +<ul> + +<li>Description: Calculates the population variance of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The population variance of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_QUADRATIC_MEAN"></a><tt>STATS_QUADRATIC_MEAN</tt></h4> + +<ul> + +<li>Description: Calculates the quadratic mean of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The quadratic mean of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_SD"></a><tt>STATS_SD</tt></h4> + +<ul> + +<li>Description: Calculates the standard deviation of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The standard deviation of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_SKEWNESS"></a><tt>STATS_SKEWNESS</tt></h4> + +<ul> + +<li>Description: Calculates the skewness of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The skewness of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_SUM"></a><tt>STATS_SUM</tt></h4> + +<ul> + +<li>Description: Calculates the sum of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The sum of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_SUM_LOGS"></a><tt>STATS_SUM_LOGS</tt></h4> + +<ul> + +<li>Description: Calculates the sum of the (natural) log of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The sum of the (natural) log of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_SUM_SQUARES"></a><tt>STATS_SUM_SQUARES</tt></h4> + +<ul> + +<li>Description: Calculates the sum of the squares of the accumulated values (or in the window if a window is used).</li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The sum of the squares of the values in the window or NaN if the statistics object is null.</li> +</ul></div> +<div class="section"> +<h4><a name="STATS_VARIANCE"></a><tt>STATS_VARIANCE</tt></h4> + +<ul> + +<li>Description: Calculates the variance of the accumulated values (or in the window if a window is used). See <a class="externalLink" href="http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics";>http://commons.apache.org/proper/commons-math/userguide/stat.html#a1.2_Descriptive_statistics</a></li> + +<li>Input: + +<ul> + +<li>stats - The Stellar statistics object</li> + </ul></li> + +<li>Returns: The variance of the values in the window or NaN if the statistics object is null.</li> +</ul></div></div> +<div class="section"> +<h3><a name="Statistical_Outlier_Detection"></a>Statistical Outlier Detection</h3> +<div class="section"> +<h4><a name="OUTLIER_MAD_STATE_MERGE"></a><tt>OUTLIER_MAD_STATE_MERGE</tt></h4> + +<ul> + +<li>Description: Update the statistical state required to compute the Median Absolute Deviation.</li> + +<li>Input: + +<ul> + +<li>[state] - A list of Median Absolute Deviation States to merge. Generally these are states across time.</li> + +<li>currentState? - The current state (optional)</li> + </ul></li> + +<li>Returns: The Median Absolute Deviation state</li> +</ul></div> +<div class="section"> +<h4><a name="OUTLIER_MAD_ADD"></a><tt>OUTLIER_MAD_ADD</tt></h4> + +<ul> + +<li>Description: Add a piece of data to the state.</li> + +<li>Input: + +<ul> + +<li>state - The MAD state</li> + +<li>value - The numeric value to add</li> + </ul></li> + +<li>Returns: The MAD state</li> +</ul></div> +<div class="section"> +<h4><a name="OUTLIER_MAD_SCORE"></a><tt>OUTLIER_MAD_SCORE</tt></h4> + +<ul> + +<li>Description: Get the modified z-score normalized by the MAD: scale * | x_i - median(X) | / MAD. See the first page of <a class="externalLink" href="http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf";>http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf</a></li> + +<li>Input: + +<ul> + +<li>state - The MAD state</li> + +<li>value - The numeric value to score</li> + +<li>scale? - Optionally the scale to use when computing the modified z-score. Default is <tt>0.6745</tt>, see the first page of <a class="externalLink" href="http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf";>http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf</a></li> + </ul></li> + +<li>Returns: The modified z-score</li> +</ul> +<p><a name="Outlier_Analysis"></a></p> +<h1>Outlier Analysis</h1> +<p>A common desire is to find anomalies in numerical data. To that end, we have some simple statistical anomaly detectors.</p></div></div></div> +<div class="section"> +<h2><a name="Median_Absolute_Deviation"></a>Median Absolute Deviation</h2> +<p>Much has been written about this robust estimator. See the first page of <a class="externalLink" href="http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf";>http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/BetterThanMAD.pdf</a> for a good coverage of the good and the bad of MAD. The usage, however is fairly straightforward:</p> + +<ul> + +<li>Gather the statistical state required to compute the MAD + +<ul> + +<li>The distribution of the values of a univariate random variable over time.</li> + +<li>The distribution of the absolute deviations of the values from the median.</li> + </ul></li> + +<li>Use this statistical state to score unseen values. The higher the score, the more unlike the previously seen data the value is.</li> +</ul> +<p>There are a couple of issues which make MAD a bit hard to compute. First, the statistical state requires computing median, which can be computationally expensive to compute exactly. To get around this, we use the OnlineStatisticalProvider to compute a sketch rather than the exact median. Secondly, the statistical state for seasonal data should be limited to a fixed, trailing window. We do this by ensuring that the MAD state is mergeable and able to be queried from within the Profiler.</p> +<div class="section"> +<h3><a name="Example"></a>Example</h3> +<p>We will create a dummy data stream of gaussian noise to illustrate how to use the MAD functionality along with the profiler to tag messages as outliers or not.</p> +<p>To do this, we will create a </p> + +<ul> + +<li>data generator</li> + +<li>parser</li> + +<li>profiler profile</li> + +<li>enrichment and threat triage</li> +</ul> +<div class="section"> +<h4><a name="Data_Generator"></a>Data Generator</h4> +<p>We can create a simple python script to generate a stream of gaussian noise at the frequency of one message per second as a python script which should be saved at <tt>~/rand_gen.py</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>#!/usr/bin/python +import random +import sys +import time +def main(): + mu = float(sys.argv[1]) + sigma = float(sys.argv[2]) + freq_s = int(sys.argv[3]) + while True: + print str(random.gauss(mu, sigma)) + sys.stdout.flush() + time.sleep(freq_s) + +if __name__ == '__main__': + main() +</pre></div></div> +<p>This script will take the following as arguments:</p> + +<ul> + +<li>The mean of the data generated</li> + +<li>The standard deviation of the data generated</li> + +<li>The frequency (in seconds) of the data generated</li> +</ul> +<p>If, however, you’d like to test a longer tailed distribution, like the student t-distribution and have numpy installed, you can use the following as <tt>~/rand_gen.py</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>#!/usr/bin/python +import random +import sys +import time +import numpy as np + +def main(): + df = float(sys.argv[1]) + freq_s = int(sys.argv[2]) + while True: + print str(np.random.standard_t(df)) + sys.stdout.flush() + time.sleep(freq_s) + +if __name__ == '__main__': + main() +</pre></div></div> +<p>This script will take the following as arguments:</p> + +<ul> + +<li>The degrees of freedom for the distribution</li> + +<li>The frequency (in seconds) of the data generated</li> +</ul></div> +<div class="section"> +<h4><a name="The_Parser"></a>The Parser</h4> +<p>We will create a parser that will take the single numbers in and create a message with a field called <tt>value</tt> in them using the <tt>CSVParser</tt>.</p> +<p>Add the following file to <tt>$METRON_HOME/config/zookeeper/parsers/mad.json</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "parserClassName" : "org.apache.metron.parsers.csv.CSVParser" + ,"sensorTopic" : "mad" + ,"parserConfig" : { + "columns" : { + "value_str" : 0 + } + } + ,"fieldTransformations" : [ + { + "transformation" : "STELLAR" + ,"output" : [ "value" ] + ,"config" : { + "value" : "TO_DOUBLE(value_str)" + } + } + ] +} +</pre></div></div></div> +<div class="section"> +<h4><a name="Enrichment_and_Threat_Intel"></a>Enrichment and Threat Intel</h4> +<p>We will set a threat triage level of <tt>10</tt> if a message generates a outlier score of more than 3.5. This cutoff will depend on your data and should be adjusted based on the assumed underlying distribution. Note that under the assumptions of normality, MAD will act as a robust estimator of the standard deviation, so the cutoff should be considered the number of standard deviations away. For other distributions, there are other interpretations which will make sense in the context of measuring the “degree different”. See <a class="externalLink" href="http://eurekastatistics.com/using-the-median-absolute-deviation-to-find-outliers/";>http://eurekastatistics.com/using-the-median-absolute-deviation-to-find-outliers/</a> for a brief discussion of this.</p> +<p>Create the following in <tt>$METRON_HOME/config/zookeeper/enrichments/mad.json</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "index": "mad", + "batchSize": 1, + "enrichment": { + "fieldMap": { + "stellar" : { + "config" : { + "parser_score" : "OUTLIER_MAD_SCORE(OUTLIER_MAD_STATE_MERGE( +PROFILE_GET( 'sketchy_mad', 'global', PROFILE_FIXED(10, 'MINUTES')) ), value)" + ,"is_alert" : "if parser_score > 3.5 then true else is_alert" + } + } + } + ,"fieldToTypeMap": { } + }, + "threatIntel": { + "fieldMap": { }, + "fieldToTypeMap": { }, + "triageConfig" : { + "riskLevelRules" : [ + { + "rule" : "parser_score > 3.5", + "score" : 10 + } + ], + "aggregator" : "MAX" + } + } +} +</pre></div></div></div> +<div class="section"> +<h4><a name="The_Profiler"></a>The Profiler</h4> +<p>We can set up the profiler to track the MAD statistical state required to compute MAD. For the purposes of this demonstration, we will configure the profiler to capture statistics on the minute mark. We will capture a global statistical state for the <tt>value</tt> field and we will look back for a 5 minute window when computing the median.</p> +<p>Create the following file at <tt>$METRON_HOME/config/zookeeper/profiler.json</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "profiles": [ + { + "profile": "sketchy_mad", + "foreach": "'global'", + "onlyif": "true", + "init" : { + "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', +'global', PROFILE_FIXED(5, 'MINUTES')))" + }, + "update": { + "s": "OUTLIER_MAD_ADD(s, value)" + }, + "result": "s" + } + ] +} +</pre></div></div> +<p>Adjust <tt>$METRON_HOME/config/zookeeper/global.json</tt> to adjust the capture duration:</p> + +<div class="source"> +<div class="source"> +<pre> "profiler.client.period.duration" : "1", + "profiler.client.period.duration.units" : "MINUTES" +</pre></div></div> +<p>Adjust <tt>$METRON_HOME/config/profiler.properties</tt> to adjust the capture duration by changing <tt>profiler.period.duration=15</tt> to <tt>profiler.period.duration=1</tt></p></div> +<div class="section"> +<h4><a name="Execute_the_Flow"></a>Execute the Flow</h4> + +<ol style="list-style-type: decimal"> + +<li> +<p>Install the elasticsearch head plugin by executing: <tt>/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head</tt></p></li> + +<li> +<p>Stopping all other parser topologies via monit</p></li> + +<li> +<p>Create the <tt>mad</tt> kafka topic by executing: <tt>/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic mad --partitions 1 --replication-factor 1</tt></p></li> + +<li> +<p>Push the modified configs by executing: <tt>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z node1:2181 -i $METRON_HOME/config/zookeeper/</tt></p></li> + +<li> +<p>Start the profiler by executing: <tt>$METRON_HOME/bin/start_profiler_topology.sh</tt></p></li> + +<li> +<p>Start the parser topology by executing: <tt>$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s mad</tt></p></li> + +<li> +<p>Ensure that the enrichment and indexing topologies are started. If not, then start those via monit or by hand.</p></li> + +<li> +<p>Generate data into kafka by executing the following for at least 10 minutes: <tt>~/rand_gen.py 0 1 1 | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic mad</tt> Note: if you chose the use the t-distribution script above, you would adjust the parameters of the <tt>rand_gen.py</tt> script accordingly.</p></li> + +<li> +<p>Stop the above with ctrl-c and send in an obvious outlier into kafka: <tt>echo "1000" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic mad</tt></p></li> +</ol> +<p>You should be able to find the outlier via the elasticsearch head plugin by searching for the messages where <tt>is_alert</tt> is <tt>true</tt>.</p></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
Added: dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/amazon-ec2/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/amazon-ec2/index.html (added) +++ dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/amazon-ec2/index.html Fri Feb 10 20:21:39 2017 @@ -0,0 +1,521 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-02-10 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170210" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Apache Metron on Amazon EC2</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> + Metron-Incubating</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Apache Metron on Amazon EC2</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-02-10</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-down"></i> + Deployment</a> + <ul class="nav nav-list"> + + <li class="active"> + + <a href="#"><i class="none"></i>Amazon-ec2</a> + </li> + + <li> + + <a href="../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"> + <i class="none"></i> + Ansible-docker</a> + </li> + + <li> + + <a href="../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"> + <i class="none"></i> + Rpm-docker</a> + </li> + + <li> + + <a href="../../metron-deployment/packer-build/index.html" title="Packer-build"> + <i class="none"></i> + Packer-build</a> + </li> + + <li> + + <a href="../../metron-deployment/roles/index.html" title="Roles"> + <i class="icon-chevron-right"></i> + Roles</a> + </li> + + <li> + + <a href="../../metron-deployment/vagrant/index.html" title="Vagrant"> + <i class="icon-chevron-right"></i> + Vagrant</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Apache Metron on Amazon EC2</h1> +<p>This project fully automates the provisioning of Apache Metron on Amazon EC2 infrastructure. Starting with only your Amazon EC2 credentials, this project will create a fully-functioning, end-to-end, multi-node cluster running Apache Metron.</p> +<p>Warning: Amazon will charge for the use of their resources when running Apache Metron. The amount will vary based on the number and size of hosts, along with current Amazon pricing structure. Be sure to stop or terminate all of the hosts instantiated by Apache Metron when not in use to avoid unnecessary charges.</p> +<div class="section"> +<h2><a name="Getting_Started"></a>Getting Started</h2> +<div class="section"> +<h3><a name="Prerequisites"></a>Prerequisites</h3> +<p>The host used to deploy Apache Metron will need the following software tools installed. The following versions are known to work as of the time of this writing, but by no means are these the only working versions.</p> + +<ul> + +<li>Ansible 2.0.0.2</li> + +<li>Python 2.7.11</li> + +<li>Maven 3.3.9</li> +</ul> +<p>Any platform that supports these tools is suitable, but the following instructions cover only macOS. The easiest means of installing these tools on a Mac is to use the excellent <a class="externalLink" href="http://brew.sh/";>Homebrew</a> project.</p> + +<ol style="list-style-type: decimal"> + +<li> +<p>Install Homebrew by running the following command in a terminal. Refer to the <a class="externalLink" href="http://brew.sh/";>Homebrew</a> home page for the latest installation instructions.</p> + +<div class="source"> +<div class="source"> +<pre> /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" +</pre></div></div></li> + +<li> +<p>With Homebrew installed, run the following command in a terminal to install all of the required tools.</p> + +<div class="source"> +<div class="source"> +<pre> brew cask install java + brew install maven git + brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/ee1273bf919a5e4e50838513a9e55ea423e1d7ce/Formula/ansible.rb + brew switch ansible 2.0.0.2 +</pre></div></div></li> + +<li> +<p>Ensure that a public SSH key is located at <tt>~/.ssh/id_rsa.pub</tt>. </p> + +<div class="source"> +<div class="source"> +<pre> $ cat ~/.ssh/id_rsa.pub + ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChv5GJxPjR39UJV7VY17ivbLVlxFrH7UHwh1Jsjem4d1eYiAtde5N2y65/HRNxWbhYli9ED8k0/MRP92ejewucEbrPNq5mytPqdC4IvZ98Ln2GbqTDwvlP3T7xa/wYFOpFsOmXXql8216wSrnrS4f3XK7ze34S6/VmY+lsBYnr3dzyj8sG/mexpJgFS/w83mWJV0e/ryf4Hd7P6DZ5fO+nmTXfKNK22ga4ctcnbZ+toYcPL+ODCh8598XCKVo97XjwF5OxN3vl1p1HHguo3cHB4H1OIaqX5mUt59gFIZcAXUME89PO6NUiZDd3RTstpf125nQVkQAHu2fvW96/f037 nick@localhost +</pre></div></div> +<p>If this file does not exist, run the following command at a terminal and accept all defaults. Only the public key, not the private key, will be uploaded to Amazon and configured on each host to enable SSH connectivity. While it is possible to create and use an alternative key those details will not be covered. </p> + +<div class="source"> +<div class="source"> +<pre> ssh-keygen -t rsa +</pre></div></div></li> +</ol></div> +<div class="section"> +<h3><a name="Amazon_Web_Services"></a>Amazon Web Services</h3> +<p>If you already have an Amazon Web Services account that you have used to deploy EC2 hosts, then you should be able to skip the next few steps.</p> + +<ol style="list-style-type: decimal"> + +<li> +<p>Head over to <a class="externalLink" href="http://aws.amazon.com/";>Amazon Web Services</a> and create an account. As part of the account creation process you will need to provide a credit card to cover any charges that may apply.</p></li> + +<li> +<p>Create a set of user credentials through <a class="externalLink" href="https://console.aws.amazon.com/iam/";>Amazon’s Identity and Access Management (IAM) </a> dashboard. On the IAM dashboard menu click “Users” and then “Create New User”. Provide a name and ensure that “Generate an access key for each user” remains checked. Download the credentials and keep them for later use.</p></li> + +<li> +<p>While still in <a class="externalLink" href="https://console.aws.amazon.com/iam/";>Amazon’s Identity and Access Management (IAM) </a> dashboard, click on the user that was previously created. Click the “Permissions” tab and then the “Attach Policy” button. Attach the following policies to the user.</p> + +<ul> + +<li>AmazonEC2FullAccess</li> + +<li>AmazonVPCFullAccess</li> + </ul></li> + +<li> +<p>Apache Metron uses the <a class="externalLink" href="https://aws.amazon.com/marketplace/pp/B00NQAYLWO";>official, open source CentOS 6</a> Amazon Machine Image (AMI). If you have never used this AMI before then you will need to accept Amazon’s terms and conditions. Navigate to the <a class="externalLink" href="https://aws.amazon.com/marketplace/pp/B00NQAYLWO";>web page for this AMI</a> and click the “Continue” button. Choose the “Manual Launch” tab then click the “Accept Software Terms” button.</p></li> +</ol> +<p>Having successfully created your Amazon Web Services account, hopefully you will find that the most difficult tasks are behind us. </p></div> +<div class="section"> +<h3><a name="Deploy_Metron"></a>Deploy Metron</h3> + +<ol style="list-style-type: decimal"> + +<li> +<p>Use the Amazon access key by exporting its values via the shell’s environment. This allows Ansible to authenticate with Amazon EC2. For example:</p> + +<div class="source"> +<div class="source"> +<pre> export AWS_ACCESS_KEY_ID="AKIAI6NRFEO27E5FFELQ" + export AWS_SECRET_ACCESS_KEY="vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed" +</pre></div></div> +<p>Notice: You must replace the access key values above with values from your own access key.</p></li> + +<li> +<p>Start the Apache Metron deployment process. When prompted provide a unique name for your Metron environment or accept the default. </p> + +<div class="source"> +<div class="source"> +<pre> $ ./run.sh + Metron Environment [metron-test]: my-metron-env + ... +</pre></div></div> +<p>The process is likely to take between 70-90 minutes. Fortunately, everything is fully automated and you should feel free to grab a coffee.</p></li> +</ol></div> +<div class="section"> +<h3><a name="Explore_Metron"></a>Explore Metron</h3> + +<ol style="list-style-type: decimal"> + +<li> +<p>After the deployment has completed successfully, a message like the following will be displayed. Navigate to the specified resources to explore your newly minted Apache Metron environment.</p> + +<div class="source"> +<div class="source"> +<pre> TASK [debug] ******************************************************************* + ok: [localhost] => { + "Success": [ + "Apache Metron deployed successfully", + " Metron @ http://ec2-52-37-255-142.us-west-2.compute.amazonaws.com:5000", + " Ambari @ http://ec2-52-37-225-202.us-west-2.compute.amazonaws.com:8080", + " Sensors @ ec2-52-37-225-202.us-west-2.compute.amazonaws.com on tap0", + "For additional information, see https://metron.incubator.apache.org/'" + ] + } +</pre></div></div></li> + +<li> +<p>Each of the provisioned hosts will be accessible from the internet. Connecting to one over SSH as the user <tt>centos</tt> will not require a password as it will authenticate with the pre-defined SSH key. </p> + +<div class="source"> +<div class="source"> +<pre> ssh cen...@ec2-52-91-215-174.compute-1.amazonaws.com +</pre></div></div></li> +</ol></div></div> +<div class="section"> +<h2><a name="Advanced_Usage"></a>Advanced Usage</h2> +<div class="section"> +<h3><a name="Multiple_Environments"></a>Multiple Environments</h3> +<p>This process can support provisioning of multiple, isolated environments. Simply change the <tt>env</tt> settings in <tt>conf/defaults.yml</tt>. For example, you might provision separate development, test, and production environments.</p> + +<div class="source"> +<div class="source"> +<pre>env: metron-test +</pre></div></div></div> +<div class="section"> +<h3><a name="Selective_Provisioning"></a>Selective Provisioning</h3> +<p>To provision only subsets of the entire Metron deployment, Ansible tags can be specified. For example, to only deploy the sensors on an Amazon EC2 environment, run the following command.</p> + +<div class="source"> +<div class="source"> +<pre>ansible-playbook -i ec2.py playbook.yml --tags "ec2,sensors" +</pre></div></div></div> +<div class="section"> +<h3><a name="Custom_SSH_Key"></a>Custom SSH Key</h3> +<p>By default, the playbook will attempt to register your public SSH key <tt>~/.ssh/id_rsa.pub</tt> with each provisioned host. This enables Ansible to communicate with each host using an SSH connection. If would prefer to use another key simply add the path to the public key file to the <tt>key_file</tt> property in <tt>conf/defaults.yml</tt>.</p> +<p>For example, generate a new SSH key for Metron that will be stored at <tt>~/.ssh/my-metron-key</tt>.</p> + +<div class="source"> +<div class="source"> +<pre>$ ssh-keygen -q -f ~/.ssh/my-metron-key +Enter passphrase (empty for no passphrase): +Enter same passphrase again: +</pre></div></div> +<p>Add the path to the newly created SSH public key to <tt>conf/defaults.yml</tt>.</p> + +<div class="source"> +<div class="source"> +<pre>key_file: ~/.ssh/metron-private-key.pub +</pre></div></div></div></div> +<div class="section"> +<h2><a name="Common_Errors"></a>Common Errors</h2> +<div class="section"> +<h3><a name="Error:_unsupported_operation_exception_custom_format_isnt_supported"></a>Error: [unsupported_operation_exception] custom format isn’t supported</h3> +<p>This error might be seen within Metron’s default dashboard in Kibana 4. This occurs when the index templates do not exist for the Snort, Bro or YAF indices in Elasticsearch. </p> +<p>The dashboard expects fields to be of a certain type. If the index templates have not been loaded correctly, the data types for the fields in these indices will be incorrect and the dashboard will display this error.</p> +<div class="section"> +<h4><a name="Solution"></a>Solution</h4> +<p>If you see this error, please report your findings by creating a JIRA or dropping an email to the Metron Users mailing list. Follow these steps to work around the problem.</p> +<p>(1) Define which Elasticsearch host to interact with. Any Elasticsearch host should work.</p> + +<div class="source"> +<div class="source"> +<pre>export ES_HOST="http://ec2-52-25-237-20.us-west-2.compute.amazonaws.com:9200" +</pre></div></div> +<p>(2) Confirm the index templates are in fact missing. </p> + +<div class="source"> +<div class="source"> +<pre>curl -s -XGET $ES_HOST/_template +</pre></div></div> +<p>(3) Manually load the index templates.</p> + +<div class="source"> +<div class="source"> +<pre>cd metron-deployment +curl -s -XPOST $ES_HOST/_template/bro_index -d @roles/metron_elasticsearch_templates/files/es_templates/bro_index.template +curl -s -XPOST $ES_HOST/_template/snort_index -d @roles/metron_elasticsearch_templates/files/es_templates/snort_index.template +curl -s -XPOST $ES_HOST/_template/yaf_index -d @roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template +</pre></div></div> +<p>(4) Delete the existing indexes. Only a new index will use the templates defined in the previous step.</p> + +<div class="source"> +<div class="source"> +<pre>curl -s -XDELETE "$ES_HOST/yaf_index*" +curl -s -XDELETE "$ES_HOST/bro_index*" +curl -s -XDELETE "$ES_HOST/snort_index*" +</pre></div></div> +<p>(5) Open up Kibana and wait for the new indexes to be created. The dashboard should now work.</p></div></div> +<div class="section"> +<h3><a name="Error:_No_handler_was_ready_to_authenticateCheck_your_credentials"></a>Error: ‘No handler was ready to authenticate…Check your credentials’</h3> + +<div class="source"> +<div class="source"> +<pre>TASK [Define keypair] ********************************************************** +failed: [localhost] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPP + r9jEFrn0yun3PYNidJ/...da...@hasselhoff.com) => {"failed": true, "item": "ssh-r + sa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPPr9jEFr... da...@hasselhoff.com", + "msg": "No handler was ready to authenticate. 1 handlers were checked. + ['HmacAuthV4Handler'] Check your credentials"} +</pre></div></div> +<div class="section"> +<h4><a name="Solution_1"></a>Solution 1</h4> +<p>This occurs when Ansible does not have the correct AWS access keys. The following commands must return a valid access key that is defined within Amazon’s <a class="externalLink" href="https://console.aws.amazon.com/iam/";>Identity and Access Management</a> console. </p> + +<div class="source"> +<div class="source"> +<pre>$ echo $AWS_ACCESS_KEY_ID +AKIAI6NRFEO27E5FFELQ + +$ echo $AWS_SECRET_ACCESS_KEY +vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed +</pre></div></div></div> +<div class="section"> +<h4><a name="Solution_2"></a>Solution 2</h4> +<p>This error can occur if you have exported the correct AWS access key, but you are using <tt>sudo</tt> to run the Ansible playbook. Do not use the <tt>sudo</tt> command when running the Ansible playbook.</p></div></div> +<div class="section"> +<h3><a name="Error:_OptInRequired:__you_need_to_accept_terms_and_subscribe"></a>Error: ‘OptInRequired: … you need to accept terms and subscribe’</h3> + +<div class="source"> +<div class="source"> +<pre>TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] ** +fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": +"Instance creation failed => OptInRequired: In order to use this AWS Marketplace +product you need to accept terms and subscribe. To do so please visit +http://aws.amazon.com/marketplace/pp?sku=6x5jmcajty9edm3f211pqjfn2"} +to retry, use: --limit @playbook.retry +</pre></div></div> +<div class="section"> +<h4><a name="Solution"></a>Solution</h4> +<p>Apache Metron uses the <a class="externalLink" href="https://aws.amazon.com/marketplace/pp?sku=6x5jmcajty9edm3f211pqjfn2";>official CentOS 6 Amazon Machine Image</a> when provisioning hosts. Amazon requires that you accept certain terms and conditions when using any Amazon Machine Image (AMI). Follow the link provided in the error message to accept the terms and conditions then re-run the playbook. </p></div></div> +<div class="section"> +<h3><a name="Error:_PendingVerification:_Your_account_is_currently_being_verified"></a>Error: ‘PendingVerification: Your account is currently being verified’</h3> + +<div class="source"> +<div class="source"> +<pre>TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] ** +fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": +"Instance creation failed => PendingVerification: Your account is currently +being verified. Verification normally takes less than 2 hours. Until your +account is verified, you may not be able to launch additional instances or +create additional volumes. If you are still receiving this message after more +than 2 hours, please let us know by writing to aws-verificat...@amazon.com. We +appreciate your patience."} +to retry, use: --limit @playbook.retry +</pre></div></div> +<div class="section"> +<h4><a name="Solution"></a>Solution</h4> +<p>This will occur if you are attempting to deploy Apache Metron using a newly created Amazon Web Services account. Follow the advice of the message and wait until Amazon’s verification process is complete. Amazon has some additional <a class="externalLink" href="http://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html";>advice for dealing with this error and more</a>.</p> + +<blockquote> +<p>Your account is pending verification. Until the verification process is complete, you may not be able to carry out requests with this account. If you have questions, contact <a class="externalLink" href="http://console.aws.amazon.com/support/home#/";>AWS Support</a>.</p> +</blockquote></div></div> +<div class="section"> +<h3><a name="Error:_Instance_creation_failed__InstanceLimitExceeded"></a>Error: ‘Instance creation failed => InstanceLimitExceeded’</h3> + +<div class="source"> +<div class="source"> +<pre>TASK [metron-test: Instantiate 3 host(s) as search,metron,ec2] ***************** +fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": +"Instance creation failed => InstanceLimitExceeded: You have requested more +instances (11) than your current instance limit of 10 allows for the specified +instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to +request an adjustment to this limit."} +to retry, use: --limit @playbook.retry +</pre></div></div> +<div class="section"> +<h4><a name="Solution"></a>Solution</h4> +<p>This will occur if Apache Metron attempts to deploy more host instances than allowed by your account. The total number of instances required for Apache Metron can be reduced by editing <tt>deployment/amazon-ec/playbook.yml</tt>. Perhaps a better alternative is to request of Amazon that this limit be increased. Amazon has some additional <a class="externalLink" href="http://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html";>advice for dealing with this error and more</a>.</p> + +<blockquote> +<p>You’ve reached the limit on the number of instances you can run concurrently. The limit depends on the instance type. For more information, see <a class="externalLink" href="http://aws.amazon.com/ec2/faqs/#How_many_instances_can_I_run_in_Amazon_EC2";>How many instances can I run in Amazon EC2</a>. If you need additional instances, complete the <a class="externalLink" href="https://console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase&limitType=service-code-ec2-instances";>Amazon EC2 Instance Request Form</a>.</p> +</blockquote></div></div> +<div class="section"> +<h3><a name="Error:_SSH_encountered_an_unknown_error_during_the_connection"></a>Error: ‘SSH encountered an unknown error during the connection’</h3> + +<div class="source"> +<div class="source"> +<pre>TASK [setup] ******************************************************************* +fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => { + "changed": false, "msg": "SSH encountered an unknown error during the + connection. We recommend you re-run the command using -vvvv, which will enable + SSH debugging output to help diagnose the issue", "unreachable": true} +</pre></div></div> +<div class="section"> +<h4><a name="Solution"></a>Solution</h4> +<p>This most often indicates that Ansible cannot connect to the host with the SSH key that it has access to. This could occur if hosts are provisioned with one SSH key, but the playbook is executed subsequently with a different SSH key. The issue can be addressed by either altering the <tt>key_file</tt> variable to point to the key that was used to provision the hosts or by simply terminating all hosts and re-running the playbook.</p></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Added: dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/index.html (added) +++ dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/index.html Fri Feb 10 20:21:39 2017 @@ -0,0 +1,434 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-02-10 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170210" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Overview</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> + <img src="../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> + Metron-Incubating</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Overview</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-02-10</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li class="active"> + + <a href="#"><i class="icon-chevron-down"></i>Deployment</a> + <ul class="nav nav-list"> + + <li> + + <a href="../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"> + <i class="none"></i> + Amazon-ec2</a> + </li> + + <li> + + <a href="../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"> + <i class="none"></i> + Ansible-docker</a> + </li> + + <li> + + <a href="../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"> + <i class="none"></i> + Rpm-docker</a> + </li> + + <li> + + <a href="../metron-deployment/packer-build/index.html" title="Packer-build"> + <i class="none"></i> + Packer-build</a> + </li> + + <li> + + <a href="../metron-deployment/roles/index.html" title="Roles"> + <i class="icon-chevron-right"></i> + Roles</a> + </li> + + <li> + + <a href="../metron-deployment/vagrant/index.html" title="Vagrant"> + <i class="icon-chevron-right"></i> + Vagrant</a> + </li> + </ul> + </li> + + <li> + + <a href="../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Overview</h1> +<p><a name="Overview"></a></p> +<p>This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster, Metron services, or both using ansible playbooks. These playbooks currently only target RHEL/CentOS 6.x operating systems.</p> +<p>In addition, an Ambari Management Pack can be built which can be deployed in conjuction with RPMs detailed in this README.</p> +<div class="section"> +<h2><a name="Prerequisites"></a>Prerequisites</h2> +<p>The following tools are required to run these scripts:</p> + +<ul> + +<li><a class="externalLink" href="https://maven.apache.org/";>Maven</a></li> + +<li><a class="externalLink" href="https://git-scm.com/";>Git</a></li> + +<li><a class="externalLink" href="http://www.ansible.com/";>Ansible</a> (version 2.0 or greater)</li> +</ul> +<p>Currently Metron must be built from source. Before running these scripts perform the following steps:</p> + +<ol style="list-style-type: decimal"> + +<li>Clone the Metron git repository with <tt>git clone g...@github.com:apache/incubator-metron.git</tt></li> + +<li>Navigate to <tt>incubator-metron</tt> and run <tt>mvn clean package</tt></li> +</ol> +<p>These scripts depend on two files for configuration:</p> + +<ul> + +<li>hosts - declares which Ansible roles will be run on which hosts</li> + +<li>group_vars/all - various configuration settings needed to install Metron</li> +</ul> +<p>Examples can be found in the <tt>incubator-metron/metron-deployment/inventory/metron_example</tt> directory and are a good starting point. Copy this directory into <tt>incubator-metron/metron-deployment/inventory/</tt> and rename it to your <tt>project_name</tt>. More information about Ansible files and directory structure can be found at <a class="externalLink" href="http://docs.ansible.com/ansible/playbooks_best_practices.html";>http://docs.ansible.com/ansible/playbooks_best_practices.html</a>.</p></div> +<div class="section"> +<h2><a name="Ambari"></a>Ambari</h2> +<p>The Ambari playbook will install a Hadoop cluster with all the services and configuration required by Metron. This section can be skipped if installing Metron on a pre-existing cluster.</p> +<p>Currently, this playbook supports building a local development cluster running on one node but options for other types of clusters will be added in the future.</p> +<div class="section"> +<h3><a name="Setting_up_your_inventory"></a>Setting up your inventory</h3> +<p>Make sure to update the hosts file in <tt>incubator-metron/metron-deployment/inventory/project_name/hosts</tt> or provide an alternate inventory file when you launch the playbooks, including the ssh user(s) and ssh keyfile location(s). These playbooks expect two host groups:</p> + +<ul> + +<li>ambari_master</li> + +<li>ambari_slaves</li> +</ul></div> +<div class="section"> +<h3><a name="Running_the_playbook"></a>Running the playbook</h3> +<p>This playbook will install the Ambari server on the ambari_master, install the ambari agents on the ambari_slaves, and create a cluster in Ambari with a blueprint for the required Metron components.</p> +<p>Navigate to <tt>incubator-metron/metron-deployment/playbooks</tt> and run: <tt>ansible-playbook -i ../inventory/project_name ambari_install.yml</tt></p></div></div> +<div class="section"> +<h2><a name="Metron"></a>Metron</h2> +<p>The Metron playbook will gather the necessary cluster settings from Ambari and install the Metron services.</p> +<div class="section"> +<h3><a name="Setting_up_your_inventory"></a>Setting up your inventory</h3> +<p>Edit the hosts file at <tt>incubator-metron/metron-deployment/inventory/project_name/hosts</tt>. Declare where which hosts the Metron services will be installed on by updating these groups:</p> + +<ul> + +<li>enrichment - submits the topology code to Storm and requires a storm client</li> + +<li>search - host where Elasticsearch will be run</li> + +<li>web - host where the Metron UI and underlying services will run</li> + +<li>sensors - host where network data will be collected and published to Kafka</li> +</ul> +<p>The Metron topologies depend on Kafka topics and HBase tables being created beforehand. Declare a host that has Kafka and HBase clients installed by updating these groups:</p> + +<ul> + +<li>metron_kafka_topics</li> + +<li>metron_hbase_tables</li> +</ul> +<p>If only installing Metron, these groups can be ignored:</p> + +<ul> + +<li>ambari_master</li> + +<li>ambari_slaves</li> +</ul></div> +<div class="section"> +<h3><a name="Configuring_group_variables"></a>Configuring group variables</h3> +<p>The Metron Ansible scripts depend on a set of variables. These variables can be found in the file at <tt>incubator-metron/metron-deployment/inventory/project_name/group_vars/all</tt>. Edit the ambari* variables to match your Ambari instance and update the java_home variable to match the java path on your hosts.</p></div> +<div class="section"> +<h3><a name="Running_the_playbook"></a>Running the playbook</h3> +<p>Navigate to <tt>incubator-metron/metron-deployment/playbooks</tt> and run: <tt>ansible-playbook -i ../inventory/project_name metron_install.yml</tt></p></div></div> +<div class="section"> +<h2><a name="Vagrant"></a>Vagrant</h2> +<p>A VagrantFile is included and will install a working version of the entire Metron stack. The following is required to run this:</p> + +<ul> + +<li><a class="externalLink" href="https://www.vagrantup.com/";>Vagrant</a></li> + +<li>Hostmanager plugin for vagrant - Run <tt>vagrant plugin install vagrant-hostmanager</tt> on the machine where Vagrant is installed</li> +</ul> +<p>Navigate to <tt>incubator-metron/metron-deployment/vagrant/full-dev-platform</tt> and run <tt>vagrant up</tt>. This also provides a good example of how to run a full end-to-end Metron install.</p></div> +<div class="section"> +<h2><a name="Ambari_Management_Pack"></a>Ambari Management Pack</h2> +<p>An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update.</p> +<p>This will set up</p> + +<ul> + +<li>Metron Parsers</li> + +<li>Enrichment</li> + +<li>Indexing</li> + +<li>GeoIP data</li> + +<li>Optional Elasticsearch</li> + +<li>Optional Kibana</li> +</ul> +<div class="section"> +<h3><a name="Prerequisites"></a>Prerequisites</h3> + +<ul> + +<li>A cluster managed by Ambari 2.4</li> + +<li>Metron RPMs available on the cluster in the /localrepo directory. See <a href="#RPM">RPM</a> for further information.</li> +</ul></div> +<div class="section"> +<h3><a name="Building_Management_Pack"></a>Building Management Pack</h3> +<p>From <tt>metron-deployment</tt> run</p> + +<div class="source"> +<div class="source"> +<pre>mvn clean package +</pre></div></div> +<p>A tar.gz that can be used with Ambari can be found at <tt>metron-deployment/packaging/ambari/metron-mpack/target/</tt></p></div> +<div class="section"> +<h3><a name="Installing_Management_Pack"></a>Installing Management Pack</h3> +<p>Before installing the mpack, update Storm’s topology.classpath in Ambari to include ‘/etc/hbase/conf:/etc/hadoop/conf’. Restart Storm service.</p> +<p>Place the mpack’s tar.gz onto the node running Ambari Server. From the command line on this node, run</p> + +<div class="source"> +<div class="source"> +<pre>ambari-server install-mpack --mpack=<mpack_location> --verbose +</pre></div></div> +<p>This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement.</p> +<p>Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default) can be set to use a <tt>file:///</tt> location</p> +<p>After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearch templates. Similar to this, a custom Kibana action to Load Template is available.</p> +<p>Another custom action is available in Ambari to import Zeppelin dashboards. See the <a href="../metron-platform/metron-indexing/index.html">metron-indexing documentation</a></p> +<div class="section"> +<h4><a name="Offline_installation"></a>Offline installation</h4> +<p>Currently there is only one point that would reach out to the internet during an install. This is the URL for the GeoIP database information.</p> +<p>The RPMs DO NOT reach out to the internet (because there is currently no hosting for them). They look on the local filesystem in <tt>/localrepo</tt>.</p></div></div> +<div class="section"> +<h3><a name="Current_Limitations"></a>Current Limitations</h3> +<p>There are a set of limitations that should be addressed based to improve the current state of the mpacks.</p> + +<ul> + +<li>There is currently no hosting for RPMs remotely. They will have to be built locally.</li> + +<li>Colocation of appropriate services should be enforced by Ambari. See [#Installing Management Pack] for more details.</li> + +<li>Storm’s topology.classpath is not updated with the Metron service install and needs to be updated separately.</li> + +<li>Several configuration parameters used when installing the Metron service could (and should) be grabbed from Ambari. Install will require them to be manually entered.</li> + +<li>Need to handle upgrading Metron</li> +</ul></div></div> +<div class="section"> +<h2><a name="RPM"></a>RPM</h2> +<p>RPMs can be built to install the components in metron-platform. These RPMs are built in a Docker container and placed into <tt>target</tt>.</p> +<p>Components in the RPMs:</p> + +<ul> + +<li>metron-common</li> + +<li>metron-data-management</li> + +<li>metron-elasticsearch</li> + +<li>metron-enrichment</li> + +<li>metron-parsers</li> + +<li>metron-pcap</li> + +<li>metron-solr</li> +</ul> +<div class="section"> +<h3><a name="Prerequisites"></a>Prerequisites</h3> + +<ul> + +<li>Docker. The image detailed in: <tt>metron-deployment/packaging/docker/rpm-docker/README.md</tt> will automatically be built (or rebuilt if necessary).</li> + +<li>Artifacts for metron-platform have been produced. E.g. <tt>mvn clean package -DskipTests</tt> in <tt>metron-platform</tt></li> +</ul></div> +<div class="section"> +<h3><a name="Building_RPMs"></a>Building RPMs</h3> +<p>From <tt>metron-deployment</tt> run</p> + +<div class="source"> +<div class="source"> +<pre>mvn clean package -Pbuild-rpms +</pre></div></div> +<p>The output RPM files will land in <tt>target/RPMS/noarch</tt>. They can be installed with the standard</p> + +<div class="source"> +<div class="source"> +<pre>rpm -i <package> +</pre></div></div></div></div> +<div class="section"> +<h2><a name="TODO"></a>TODO</h2> + +<ul> + +<li>Support Ubuntu deployments</li> +</ul></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
