Added: dev/incubator/metron/0.3.1-RC5-incubating/book-site/metron-platform/metron-data-management/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC5-incubating/book-site/metron-platform/metron-data-management/index.html (added) +++ dev/incubator/metron/0.3.1-RC5-incubating/book-site/metron-platform/metron-data-management/index.html Thu Feb 23 16:12:27 2017 @@ -0,0 +1,1014 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-02-23 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Resource Data Management</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> + Metron-Incubating</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Resource Data Management</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-down"></i> + Platform</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-platform/metron-api/index.html" title="Api"> + <i class="none"></i> + Api</a> + </li> + + <li> + + <a href="../../metron-platform/metron-common/index.html" title="Common"> + <i class="none"></i> + Common</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Data-management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"> + <i class="none"></i> + Enrichment</a> + </li> + + <li> + + <a href="../../metron-platform/metron-indexing/index.html" title="Indexing"> + <i class="none"></i> + Indexing</a> + </li> + + <li> + + <a href="../../metron-platform/metron-management/index.html" title="Management"> + <i class="none"></i> + Management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> + <i class="none"></i> + Parsers</a> + </li> + + <li> + + <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"> + <i class="none"></i> + Pcap-backend</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Resource Data Management</h1> +<p><a name="Resource_Data_Management"></a></p> +<p>This project is a collection of classes to assist with loading of various enrichment and threat intelligence sources into Metron.</p> +<div class="section"> +<h2><a name="Simple_HBase_EnrichmentsThreat_Intelligence"></a>Simple HBase Enrichments/Threat Intelligence</h2> +<p>The vast majority of enrichments and threat intelligence processing tend toward the following pattern:</p> + +<ul> + +<li>Take a field</li> + +<li>Look up the field in a key/value store</li> + +<li>If the key exists, then either it’s a threat to be alerted or it should be enriched with the value associated with the key.</li> +</ul> +<p>As such, we have created this capability as a default threat intel and enrichment adapter. The basic primitive for simple enrichments and threat intelligence sources is a complex key containing the following:</p> + +<ul> + +<li>Type : The type of threat intel or enrichment (e.g. malicious_ip)</li> + +<li>Indicator : The indicator in question</li> + +<li>Value : The value to associate with the type, indicator pair. This is a JSON map.</li> +</ul> +<p>At present, all of the dataloads utilities function by converting raw data sources to this primitive key (type, indicator) and value to be placed in HBase.</p> +<p>In the case of threat intel, a hit on the threat intel table will result in:</p> + +<ul> + +<li>The <tt>is_alert</tt> field being set to <tt>true</tt> in the index</li> + +<li>A field named <tt>threatintels.hbaseThreatIntel.$field.$threatintel_type</tt> is set to <tt>alert</tt> + +<ul> + +<li><tt>$field</tt> is the field in the original document that was a match (e.g. <tt>src_ip_addr</tt>)</li> + +<li><tt>$threatintel_type</tt> is the type of threat intel imported (defined in the Extractor configuration below).</li> + </ul></li> +</ul> +<p>In the case of simple hbase enrichment, a hit on the enrichments table will result in the following new field for each key in the value:<tt>enrichments.hbaseEnrichment.$field.$enrichment_type.$key</tt> </p> + +<ul> + +<li><tt>$field</tt> is the field in the original document that was a match (e.g. <tt>src_ip_addr</tt>)</li> + +<li><tt>$enrichment_type</tt> is the type of enrichment imported (defined in the Extractor configuration below).</li> + +<li><tt>$key</tt> is a key in the JSON map associated with the row in HBase.</li> +</ul> +<p>For instance, in the situation where we had the following very silly key/value in HBase in the enrichment table:</p> + +<ul> + +<li>indicator: <tt>127.0.0.1</tt></li> + +<li>type : <tt>important_addresses</tt></li> + +<li>value: <tt>{ "name" : "localhost", "location" : "home" }</tt></li> +</ul> +<p>If we had a document whose <tt>ip_src_addr</tt> came through with a value of <tt>127.0.0.1</tt>, we would have the following fields added to the indexed document:</p> + +<ul> + +<li><tt>enrichments.hbaseEnrichment.ip_src_addr.important_addresses.name</tt> : <tt>localhost</tt></li> + +<li><tt>enrichments.hbaseEnrichment.ip_src_addr.important_addresses.location</tt> : <tt>home</tt></li> +</ul></div> +<div class="section"> +<h2><a name="Extractor_Framework"></a>Extractor Framework</h2> +<p>For the purpose of ingesting data of a variety of formats, we have created an Extractor framework which allows for common data formats to be interpreted as enrichment or threat intelligence sources. The formats supported at present are:</p> + +<ul> + +<li>CSV (both threat intel and enrichment)</li> + +<li>STIX (threat intel only)</li> + +<li>Custom (pass your own class)</li> +</ul> +<p>All of the current utilities take a JSON file to configure how to interpret input data. This JSON describes the type of data and the schema if necessary for the data if it is not fixed (as in STIX, e.g.).</p> +<div class="section"> +<h3><a name="CSV_Extractor"></a>CSV Extractor</h3> +<p>Consider the following example configuration file which describes how to process a CSV file.</p> + +<div class="source"> +<div class="source"> +<pre>{ + "config" : { + "columns" : { + "ip" : 0 + ,"source" : 2 + } + ,"indicator_column" : "ip" + ,"type" : "malicious_ip" + ,"separator" : "," + } + ,"extractor" : "CSV" +} +</pre></div></div> +<p>In this example, we have instructed the extractor of the schema (i.e. the columns field), two columns at the first and third position. We have indicated that the <tt>ip</tt> column is the indicator type and that the enrichment type is named <tt>malicious_ip</tt>. We have also indicated that the extractor to use is the CSV Extractor. The other option is the STIX extractor or a fully qualified classname for your own extractor.</p> +<p>The meta column values will show up in the value in HBase because it is called out as a non-indicator column. The key for the value will be ‘meta’. For instance, given an input string of <tt>123.45.123.12,something,the grapevine</tt>, the following key, value would be extracted:</p> + +<ul> + +<li>Indicator : <tt>123.45.123.12</tt></li> + +<li>Type : <tt>malicious_ip</tt></li> + +<li>Value : <tt>{ "ip" : "123.45.123.12", "source" : "the grapevine" }</tt></li> +</ul></div> +<div class="section"> +<h3><a name="STIX_Extractor"></a>STIX Extractor</h3> +<p>Consider the following config for importing STIX documents. This is a threat intelligence interchange format, so it is particularly relevant and attractive data to import for our purposes. Because STIX is a standard format, there is no need to specify the schema or how to interpret the documents.</p> +<p>We support a subset of STIX messages for importation:</p> + +<table border="0" class="table table-striped"> + <thead> + +<tr class="a"> + +<th>STIX Type </th> + +<th>Specific Type </th> + +<th>Enrichment Type Name </th> + </tr> + </thead> + <tbody> + +<tr class="b"> + +<td>Address </td> + +<td>IPV_4_ADDR </td> + +<td>address:IPV_4_ADDR </td> + </tr> + +<tr class="a"> + +<td>Address </td> + +<td>IPV_6_ADDR </td> + +<td>address:IPV_6_ADDR </td> + </tr> + +<tr class="b"> + +<td>Address </td> + +<td>E_MAIL </td> + +<td>address:E_MAIL </td> + </tr> + +<tr class="a"> + +<td>Address </td> + +<td>MAC </td> + +<td>address:MAC </td> + </tr> + +<tr class="b"> + +<td>Domain </td> + +<td>FQDN </td> + +<td>domain:FQDN </td> + </tr> + +<tr class="a"> + +<td>Hostname </td> + +<td> </td> + +<td>hostname </td> + </tr> + </tbody> +</table> +<p>NOTE: The enrichment type will be used as the type above.</p> +<p>Consider the following configuration for an Extractor</p> + +<div class="source"> +<div class="source"> +<pre>{ + "config" : { + "stix_address_categories" : "IPV_4_ADDR" + } + ,"extractor" : "STIX" +} +</pre></div></div> +<p>In here, we’re configuring the STIX extractor to load from a series of STIX files, however we only want to bring in IPv4 addresses from the set of all possible addresses. Note that if no categories are specified for import, all are assumed. Also, only address and domain types allow filtering via <tt>stix_address_categories</tt> and <tt>stix_domain_categories</tt> config parameters.</p></div> +<div class="section"> +<h3><a name="Common_Extractor_Properties"></a>Common Extractor Properties</h3> +<p>Users also have the ability to transform and filter enrichment and threat intel data using Stellar as it is loaded into HBase. This feature is available to all extractor types.</p> +<p>As an example, we will be providing a CSV list of top domains as an enrichment and filtering the value metadata, as well as the indicator column, with Stellar expressions.</p> + +<div class="source"> +<div class="source"> +<pre>{ + "config" : { + "zk_quorum" : "node1:2181", + "columns" : { + "rank" : 0, + "domain" : 1 + }, + "value_transform" : { + "domain" : "DOMAIN_REMOVE_TLD(domain)" + }, + "value_filter" : "LENGTH(domain) > 0", + "indicator_column" : "domain", + "indicator_transform" : { + "indicator" : "DOMAIN_REMOVE_TLD(indicator)" + }, + "indicator_filter" : "LENGTH(indicator) > 0", + "type" : "top_domains", + "separator" : "," + }, + "extractor" : "CSV" +} +</pre></div></div> +<p>There are 2 property maps that work with full Stellar expressions, and 2 properties that will work with Stellar predicates.</p> + +<table border="0" class="table table-striped"> + <thead> + +<tr class="a"> + +<th>Property </th> + +<th>Description </th> + </tr> + </thead> + <tbody> + +<tr class="b"> + +<td>value_transform </td> + +<td>Transform fields defined in the “columns” mapping with Stellar transformations. New keys introduced in the transform will be added to the key metadata. </td> + </tr> + +<tr class="a"> + +<td>value_filter </td> + +<td>Allows additional filtering with Stellar predicates based on results from the value transformations. In this example, records whose domain property is empty after removing the TLD will be omitted. </td> + </tr> + +<tr class="b"> + +<td>indicator_transform </td> + +<td>Transform the indicator column independent of the value transformations. You can refer to the original indicator value by using “indicator” as the variable name, as shown in the example above. In addition, if you prefer to piggyback your transformations, you can refer to the variable “domain”, which will allow your indicator transforms to inherit transformations done to this value during the value transformations. </td> + </tr> + +<tr class="a"> + +<td>indicator_filter </td> + +<td>Allows additional filtering with Stellar predicates based on results from the value transformations. In this example, records whose indicator value is empty after removing the TLD will be omitted. </td> + </tr> + </tbody> +</table> +<p>top-list.csv</p> + +<div class="source"> +<div class="source"> +<pre>1,google.com +2,youtube.com +... +</pre></div></div> +<p>Running a file import with the above data and extractor configuration would result in the following 2 extracted data records:</p> + +<table border="0" class="table table-striped"> + <thead> + +<tr class="a"> + +<th>Indicator </th> + +<th>Type </th> + +<th>Value </th> + </tr> + </thead> + <tbody> + +<tr class="b"> + +<td>google </td> + +<td>top_domains </td> + +<td>{ “rank” : “1”, “domain” : “google” } </td> + </tr> + +<tr class="a"> + +<td>yahoo </td> + +<td>top_domains </td> + +<td>{ “rank” : “2”, “domain” : “yahoo” } </td> + </tr> + </tbody> +</table> +<p>Similar to the parser framework, providing a Zookeeper quorum via the zk_quorum property will enable Stellar to access properties that reside in the global config. Expanding on our example above, if the global config looks as follows:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "global_property" : "metron-ftw" +} +</pre></div></div> +<p>And we expand our value_tranform:</p> + +<div class="source"> +<div class="source"> +<pre>... + "value_transform" : { + "domain" : "DOMAIN_REMOVE_TLD(domain)", + "a-new-prop" : "global_property" + }, +... + +</pre></div></div> +<p>The resulting value data would look like the following:</p> + +<table border="0" class="table table-striped"> + <thead> + +<tr class="a"> + +<th>Indicator </th> + +<th>Type </th> + +<th>Value </th> + </tr> + </thead> + <tbody> + +<tr class="b"> + +<td>google </td> + +<td>top_domains </td> + +<td>{ “rank” : “1”, “domain” : “google”, “a-new-prop” : “metron-ftw” } </td> + </tr> + +<tr class="a"> + +<td>yahoo </td> + +<td>top_domains </td> + +<td>{ “rank” : “2”, “domain” : “yahoo”, “a-new-prop” : “metron-ftw” } </td> + </tr> + </tbody> +</table></div></div> +<div class="section"> +<h2><a name="Enrichment_Config"></a>Enrichment Config</h2> +<p>In order to automatically add new enrichment and threat intel types to existing, running enrichment topologies, you will need to add new fields and new types to the zookeeper configuration. A convenience parameter has been made to assist in this when doing an import. Namely, you can specify the enrichment configs and how they associate with the fields of the documents flowing through the enrichment topology.</p> +<p>Consider the following Enrichment Configuration JSON. This one is for a threat intelligence type:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "zkQuorum" : "localhost:2181" + ,"sensorToFieldList" : { + "bro" : { + "type" : "THREAT_INTEL" + ,"fieldToEnrichmentTypes" : { + "ip_src_addr" : [ "malicious_ip" ] + ,"ip_dst_addr" : [ "malicious_ip" ] + } + } + } +} +</pre></div></div> +<p>We have to specify the following:</p> + +<ul> + +<li>The zookeeper quorum which holds the cluster configuration</li> + +<li>The mapping between the fields in the enriched documents and the enrichment types.</li> +</ul> +<p>This configuration allows the ingestion tools to update zookeeper post-ingestion so that the enrichment topology can take advantage immediately of the new type.</p></div> +<div class="section"> +<h2><a name="Loading_Utilities"></a>Loading Utilities</h2> +<p>The two configurations above are used in the three separate ingestion tools:</p> + +<ul> + +<li>Taxii Loader</li> + +<li>Bulk load from HDFS via MapReduce</li> + +<li>Flat File ingestion</li> +</ul> +<div class="section"> +<h3><a name="Taxii_Loader"></a>Taxii Loader</h3> +<p>The shell script <tt>$METRON_HOME/bin/threatintel_taxii_load.sh</tt> can be used to poll a Taxii server for STIX documents and ingest them into HBase.<br />It is quite common for this Taxii server to be an aggregation server such as Soltra Edge.</p> +<p>In addition to the Enrichment and Extractor configs described above, this loader requires a configuration file describing the connection information to the Taxii server. An illustrative example of such a configuration file is:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "endpoint" : "http://localhost:8282/taxii-discovery-service" + ,"type" : "DISCOVER" + ,"collection" : "guest.Abuse_ch" + ,"table" : "threat_intel" + ,"columnFamily" : "cf" + ,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ] +} +</pre></div></div> +<p>As you can see, we are specifying the following information:</p> + +<ul> + +<li>endpoint : The URL of the endpoint</li> + +<li>type : <tt>POLL</tt> or <tt>DISCOVER</tt> depending on the endpoint.</li> + +<li>collection : The Taxii collection to ingest</li> + +<li>table : The HBase table to import into</li> + +<li>columnFamily : The column family to import into</li> + +<li>allowedIndicatorTypes : an array of acceptable threat intel types (see the “Enrichment Type Name” column of the Stix table above for the possibilities).</li> +</ul> +<p>The parameters for the utility are as follows:</p> + +<table border="0" class="table table-striped"> + <thead> + +<tr class="a"> + +<th>Short Code </th> + +<th>Long Code </th> + +<th>Is Required? </th> + +<th>Description </th> + </tr> + </thead> + <tbody> + +<tr class="b"> + +<td>-h </td> + +<td> </td> + +<td>No </td> + +<td>Generate the help screen/set of options </td> + </tr> + +<tr class="a"> + +<td>-e </td> + +<td>–extractor_config </td> + +<td>Yes </td> + +<td>JSON Document describing the extractor for this input data source </td> + </tr> + +<tr class="b"> + +<td>-c </td> + +<td>–taxii_connection_config </td> + +<td>Yes </td> + +<td>The JSON config file to configure the connection </td> + </tr> + +<tr class="a"> + +<td>-p </td> + +<td>–time_between_polls </td> + +<td>No </td> + +<td>The time between polling the Taxii server in milliseconds. (default: 1 hour) </td> + </tr> + +<tr class="b"> + +<td>-b </td> + +<td>–begin_time </td> + +<td>No </td> + +<td>Start time to poll the Taxii server (all data from that point will be gathered in the first pull). The format for the date is yyyy-MM-dd HH:mm:ss </td> + </tr> + +<tr class="a"> + +<td>-l </td> + +<td>–log4j </td> + +<td>No </td> + +<td>The Log4j Properties to load </td> + </tr> + +<tr class="b"> + +<td>-n </td> + +<td>–enrichment_config </td> + +<td>No </td> + +<td>The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified. </td> + </tr> + </tbody> +</table></div> +<div class="section"> +<h3><a name="Flatfile_Loader"></a>Flatfile Loader</h3> +<p>The shell script <tt>$METRON_HOME/bin/flatfile_loader.sh</tt> will read data from local disk, HDFS or URLs and load the enrichment or threat intel data into an HBase table.<br />Note: This utility works for enrichment as well as threat intel due to the underlying infrastructure being the same.</p> +<p>One special thing to note here is that there is a special configuration parameter to the Extractor config that is only considered during this loader:</p> + +<ul> + +<li>inputFormat : This specifies how to consider the data. The two implementations are <tt>BY_LINE</tt> and <tt>WHOLE_FILE</tt>.</li> +</ul> +<p>The default is <tt>BY_LINE</tt>, which makes sense for a list of CSVs where each line indicates a unit of information which can be imported. However, if you are importing a set of STIX documents, then you want each document to be considered as input to the Extractor.</p> +<p>The parameters for the utility are as follows:</p> + +<table border="0" class="table table-striped"> + <thead> + +<tr class="a"> + +<th>Short Code </th> + +<th>Long Code </th> + +<th>Is Required? </th> + +<th>Description </th> + +<th> </th> + </tr> + </thead> + <tbody> + +<tr class="b"> + +<td>-h </td> + +<td> </td> + +<td>No </td> + +<td>Generate the help screen/set of options </td> + +<td> </td> + </tr> + +<tr class="a"> + +<td>-q </td> + +<td>–quiet </td> + +<td>No </td> + +<td>Do not update progress</td> + </tr> + +<tr class="b"> + +<td>-e </td> + +<td>–extractor_config </td> + +<td>Yes </td> + +<td>JSON Document describing the extractor for this input data source </td> + +<td> </td> + </tr> + +<tr class="a"> + +<td>-m </td> + +<td>–import_mode </td> + +<td>No </td> + +<td>The Import mode to use: LOCAL, MR. Default: LOCAL </td> + +<td> </td> + </tr> + +<tr class="b"> + +<td>-t </td> + +<td>–hbase_table </td> + +<td>Yes </td> + +<td>The HBase table to import into </td> + +<td> </td> + </tr> + +<tr class="a"> + +<td>-c </td> + +<td>–hbase_cf </td> + +<td>Yes </td> + +<td>The HBase table column family to import into </td> + +<td> </td> + </tr> + +<tr class="b"> + +<td>-i </td> + +<td>–input </td> + +<td>Yes </td> + +<td>The input data location on local disk. If this is a file, then that file will be loaded. If this is a directory, then the files will be loaded recursively under that directory. </td> + +<td> </td> + </tr> + +<tr class="a"> + +<td>-l </td> + +<td>–log4j </td> + +<td>No </td> + +<td>The log4j properties file to load </td> + +<td> </td> + </tr> + +<tr class="b"> + +<td>-n </td> + +<td>–enrichment_config </td> + +<td>No </td> + +<td>The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified. </td> + +<td> </td> + </tr> + +<tr class="a"> + +<td>-p </td> + +<td>–threads </td> + +<td>No </td> + +<td>The number of threads to use when extracting data. The default is the number of cores. </td> + +<td> </td> + </tr> + +<tr class="b"> + +<td>-b </td> + +<td>–batchSize </td> + +<td>No </td> + +<td>The batch size to use for HBase puts </td> + +<td> </td> + </tr> + </tbody> +</table></div> +<div class="section"> +<h3><a name="GeoLite2_Loader"></a>GeoLite2 Loader</h3> +<p>The shell script <tt>$METRON_HOME/bin/geo_enrichment_load.sh</tt> will retrieve MaxMind GeoLite2 data and load data into HDFS, and update the configuration.</p> +<p>THIS SCRIPT WILL NOT UPDATE AMBARI’S GLOBAL.JSON, JUST THE ZK CONFIGS. CHANGES WILL GO INTO EFFECT, BUT WILL NOT PERSIST PAST AN AMBARI RESTART UNTIL UPDATED THERE.</p> +<p>The parameters for the utility are as follows:</p> + +<table border="0" class="table table-striped"> + <thead> + +<tr class="a"> + +<th>Short Code </th> + +<th>Long Code </th> + +<th>Is Required? </th> + +<th>Description </th> + </tr> + </thead> + <tbody> + +<tr class="b"> + +<td>-h </td> + +<td> </td> + +<td>No </td> + +<td>Generate the help screen/set of options </td> + </tr> + +<tr class="a"> + +<td>-g </td> + +<td>–geo_url </td> + +<td>No </td> + +<td>GeoIP URL - defaults to <a class="externalLink" href="http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz";>http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz</a></td> + </tr> + +<tr class="b"> + +<td>-r </td> + +<td>–remote_dir </td> + +<td>No </td> + +<td>HDFS directory to land formatted GeoIP file - defaults to /apps/metron/geo/<epoch millis>/</td> + </tr> + +<tr class="a"> + +<td>-t </td> + +<td>–tmp_dir </td> + +<td>No </td> + +<td>Directory for landing the temporary GeoIP data - defaults to /tmp</td> + </tr> + +<tr class="b"> + +<td>-z </td> + +<td>–zk_quorum </td> + +<td>Yes </td> + +<td>Zookeeper Quorum URL (zk1:port,zk2:port,…)</td> + </tr> + </tbody> +</table></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
Added: dev/incubator/metron/0.3.1-RC5-incubating/book-site/metron-platform/metron-enrichment/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC5-incubating/book-site/metron-platform/metron-enrichment/index.html (added) +++ dev/incubator/metron/0.3.1-RC5-incubating/book-site/metron-platform/metron-enrichment/index.html Thu Feb 23 16:12:27 2017 @@ -0,0 +1,670 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-02-23 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Enrichment</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> + Metron-Incubating</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Enrichment</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-down"></i> + Platform</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-platform/metron-api/index.html" title="Api"> + <i class="none"></i> + Api</a> + </li> + + <li> + + <a href="../../metron-platform/metron-common/index.html" title="Common"> + <i class="none"></i> + Common</a> + </li> + + <li> + + <a href="../../metron-platform/metron-data-management/index.html" title="Data-management"> + <i class="none"></i> + Data-management</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Enrichment</a> + </li> + + <li> + + <a href="../../metron-platform/metron-indexing/index.html" title="Indexing"> + <i class="none"></i> + Indexing</a> + </li> + + <li> + + <a href="../../metron-platform/metron-management/index.html" title="Management"> + <i class="none"></i> + Management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> + <i class="none"></i> + Parsers</a> + </li> + + <li> + + <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"> + <i class="none"></i> + Pcap-backend</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Enrichment</h1> +<p><a name="Enrichment"></a></p> +<div class="section"> +<h2><a name="Introduction"></a>Introduction</h2> +<p>The <tt>enrichment</tt> topology is a topology dedicated to taking the data from the parsing topologies that have been normalized into the Metron data format (e.g. a JSON Map structure with <tt>original_message</tt> and <tt>timestamp</tt>) and </p> + +<ul> + +<li>Enriching messages with external data from data stores (e.g. hbase) by adding new fields based on existing fields in the messages.</li> + +<li>Marking messages as threats based on data in external data stores</li> + +<li>Marking threat alerts with a numeric triage level based on a set of Stellar rules.</li> +</ul></div> +<div class="section"> +<h2><a name="Enrichment_Architecture"></a>Enrichment Architecture</h2> +<p><img src="../../images/enrichment_arch.png" alt="Architecture" /></p></div> +<div class="section"> +<h2><a name="Enrichment_Configuration"></a>Enrichment Configuration</h2> +<p>The configuration for the <tt>enrichment</tt> topology, the topology primarily responsible for enrichment and threat intelligence enrichment, is defined by JSON documents stored in zookeeper.</p> +<p>There are two types of configurations at the moment, <tt>global</tt> and <tt>sensor</tt> specific. </p></div> +<div class="section"> +<h2><a name="Global_Configuration"></a>Global Configuration</h2> +<p>See the “<a href="../metron-common/index.html">Global Configuration</a>” section.</p></div> +<div class="section"> +<h2><a name="Sensor_Enrichment_Configuration"></a>Sensor Enrichment Configuration</h2> +<p>The sensor specific configuration is intended to configure the individual enrichments and threat intelligence enrichments for a given sensor type (e.g. <tt>snort</tt>).</p> +<p>Just like the global config, the format is a JSON stored in zookeeper. The configuration is a complex JSON object with the following top level fields:</p> + +<ul> + +<li><tt>enrichment</tt> : A complex JSON object representing the configuration of the enrichments</li> + +<li><tt>threatIntel</tt> : A complex JSON object representing the configuration of the threat intelligence enrichments</li> +</ul> +<div class="section"> +<h3><a name="The_enrichment_Configuration"></a>The <tt>enrichment</tt> Configuration</h3> + +<table border="0" class="table table-striped"> + <thead> + +<tr class="a"> + +<th>Field </th> + +<th>Description </th> + +<th>Example </th> + </tr> + </thead> + <tbody> + +<tr class="b"> + +<td><tt>fieldToTypeMap</tt> </td> + +<td>In the case of a simple HBase enrichment (i.e. a key/value lookup), the mapping between fields and the enrichment types associated with those fields must be known. This enrichment type is used as part of the HBase key. Note: applies to hbaseEnrichment only. </td> + +<td><tt>"fieldToTypeMap" : { "ip_src_addr" : [ "asset_enrichment" ] }</tt> </td> + </tr> + +<tr class="a"> + +<td><tt>fieldMap</tt> </td> + +<td>The map of enrichment bolts names to configuration handlers which know how to split the message up. The simplest of which is just a list of fields. More complex examples would be the stellar enrichment which provides stellar statements. Each field listed in the array arg is sent to the enrichment referenced in the key. Cardinality of fields to enrichments is many-to-many. </td> + +<td><tt>"fieldMap": {"hbaseEnrichment": ["ip_src_addr","ip_dst_addr"]}</tt> </td> + </tr> + +<tr class="b"> + +<td><tt>config</tt> </td> + +<td>The general configuration for the enrichment </td> + +<td><tt>"config": {"typeToColumnFamily": { "asset_enrichment" : "cf" } }</tt> </td> + </tr> + </tbody> +</table> +<p>The <tt>config</tt> map is intended to house enrichment specific configuration. For instance, for the <tt>hbaseEnrichment</tt>, the mappings between the enrichment types to the column families is specified.</p> +<p>The <tt>fieldMap</tt>contents are of interest because they contain the routing and configuration information for the enrichments. When we say ‘routing’, we mean how the messages get split up and sent to the enrichment adapter bolts. The simplest, by far, is just providing a simple list as in</p> + +<div class="source"> +<div class="source"> +<pre> "fieldMap": { + "geo": [ + "ip_src_addr", + "ip_dst_addr" + ], + "host": [ + "ip_src_addr", + "ip_dst_addr" + ], + "hbaseEnrichment": [ + "ip_src_addr", + "ip_dst_addr" + ] + } +</pre></div></div> +<p>Based on this sample config, both ip_src_addr and ip_dst_addr will go to the <tt>geo</tt>, <tt>host</tt>, and <tt>hbaseEnrichment</tt> adapter bolts. For the <tt>geo</tt>, <tt>host</tt> and <tt>hbaseEnrichment</tt>, this is sufficient. However, more complex enrichments may contain their own configuration. Currently, the <tt>stellar</tt> enrichment requires a more complex configuration, such as:</p> + +<div class="source"> +<div class="source"> +<pre> "fieldMap": { + ... + "stellar" : { + "config" : { + "numeric" : { + "foo": "1 + 1" + } + ,"ALL_CAPS" : "TO_UPPER(source.type)" + } + } + } +</pre></div></div> +<p>Whereas the simpler enrichments just need a set of fields explicitly stated so they can be separated from the message and sent to the enrichment adapter bolt for enrichment and ultimately joined back in the join bolt, the stellar enrichment has its set of required fields implicitly stated through usage. For instance, if your stellar statement references a field, it should be included and if not, then it should not be included. We did not want to require users to make explicit the implicit.</p> +<p>The other way in which the stellar enrichment is somewhat more complex is in how the statements are executed. In the general purpose case for a list of fields, those fields are used to create a message to send to the enrichment adapter bolt and that bolt’s worker will handle the fields one by one in serial for a given message. For stellar enrichment, we wanted to have a more complex design so that users could specify the groups of stellar statements sent to the same worker in the same message (and thus executed sequentially). Consider the following configuration:</p> + +<div class="source"> +<div class="source"> +<pre> "fieldMap": { + "stellar" : { + "config" : { + "numeric" : { + "foo": "1 + 1" + "bar" : TO_LOWER(source.type)" + } + ,"text" : { + "ALL_CAPS" : "TO_UPPER(source.type)" + } + } + } + } +</pre></div></div> +<p>We have a group called <tt>numeric</tt> whose stellar statements will be executed sequentially. In parallel to that, we have the group of stellar statements under the group <tt>text</tt> executing. The intent here is to allow you to not force higher latency operations to be done sequentially. You can use any name for your groupings you like. Be aware that the configuration is a map and duplicate configuration keys’ values are not combined, so the duplicate configuration value will be overwritten.</p></div> +<div class="section"> +<h3><a name="The_threatIntel_Configuration"></a>The <tt>threatIntel</tt> Configuration</h3> + +<table border="0" class="table table-striped"> + <thead> + +<tr class="a"> + +<th>Field </th> + +<th>Description </th> + +<th>Example </th> + </tr> + </thead> + <tbody> + +<tr class="b"> + +<td><tt>fieldToTypeMap</tt> </td> + +<td>In the case of a simple HBase threat intel enrichment (i.e. a key/value lookup), the mapping between fields and the enrichment types associated with those fields must be known. This enrichment type is used as part of the HBase key. Note: applies to hbaseThreatIntel only. </td> + +<td><tt>"fieldToTypeMap" : { "ip_src_addr" : [ "malicious_ips" ] }</tt> </td> + </tr> + +<tr class="a"> + +<td><tt>fieldMap</tt> </td> + +<td>The map of threat intel enrichment bolts names to fields in the JSON messages. Each field is sent to the threat intel enrichment bolt referenced in the key. Each field listed in the array arg is sent to the enrichment referenced in the key. Cardinality of fields to enrichments is many-to-many. </td> + +<td><tt>"fieldMap": {"hbaseThreatIntel": ["ip_src_addr","ip_dst_addr"]}</tt> </td> + </tr> + +<tr class="b"> + +<td><tt>triageConfig</tt> </td> + +<td>The configuration of the threat triage scorer. In the situation where a threat is detected, a score is assigned to the message and embedded in the indexed message. </td> + +<td><tt>"riskLevelRules" : { "IN_SUBNET(ip_dst_addr, '192.168.0.0/24')" : 10 }</tt> </td> + </tr> + +<tr class="a"> + +<td><tt>config</tt> </td> + +<td>The general configuration for the Threat Intel </td> + +<td><tt>"config": {"typeToColumnFamily": { "malicious_ips","cf" } }</tt> </td> + </tr> + </tbody> +</table> +<p>The <tt>config</tt> map is intended to house threat intel specific configuration. For instance, for the <tt>hbaseThreatIntel</tt> threat intel adapter, the mappings between the enrichment types to the column families is specified.</p> +<p>The <tt>triageConfig</tt> field is also a complex field and it bears some description:</p> + +<table border="0" class="table table-striped"> + <thead> + +<tr class="a"> + +<th>Field </th> + +<th>Description </th> + +<th>Example </th> + </tr> + </thead> + <tbody> + +<tr class="b"> + +<td><tt>riskLevelRules</tt> </td> + +<td>This is a list of rules (represented as Stellar expressions) associated with scores with optional names and comments </td> + +<td>see below</td> + </tr> + +<tr class="a"> + +<td><tt>aggregator</tt> </td> + +<td>An aggregation function that takes all non-zero scores representing the matching queries from <tt>riskLevelRules</tt> and aggregates them into a single score. </td> + +<td><tt>"MAX"</tt> </td> + </tr> + </tbody> +</table> +<p>A risk level rule is of the following format:</p> + +<ul> + +<li><tt>name</tt> : The name of the threat triage rule</li> + +<li><tt>comment</tt> : A comment describing the rule</li> + +<li><tt>rule</tt> : The rule, represented as a Stellar statement</li> + +<li><tt>score</tt> : Associated threat triage score for the rule</li> +</ul> +<p>An example of a rule is as follows:</p> + +<div class="source"> +<div class="source"> +<pre> "riskLevelRules" : [ + { + "name" : "is internal" + , "comment" : "determines if the destination is internal." + , rule" : "IN_SUBNET(ip_dst_addr, '192.168.0.0/24')" + , "score" : 10 + } + ] +</pre></div></div> +<p>The supported aggregation functions are:</p> + +<ul> + +<li><tt>MAX</tt> : The max of all of the associated values for matching queries</li> + +<li><tt>MIN</tt> : The min of all of the associated values for matching queries</li> + +<li><tt>MEAN</tt> : The mean of all of the associated values for matching queries</li> + +<li><tt>POSITIVE_MEAN</tt> : The mean of the positive associated values for the matching queries.</li> +</ul></div> +<div class="section"> +<h3><a name="Example_Configuration"></a>Example Configuration</h3> +<p>An example configuration for the YAF sensor is as follows:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "enrichment": { + "fieldMap": { + "geo": [ + "ip_src_addr", + "ip_dst_addr" + ], + "host": [ + "ip_src_addr", + "ip_dst_addr" + ], + "hbaseEnrichment": [ + "ip_src_addr", + "ip_dst_addr" + ] + } + ,"fieldToTypeMap": { + "ip_src_addr": [ + "playful_classification" + ], + "ip_dst_addr": [ + "playful_classification" + ] + } + }, + "threatIntel": { + "fieldMap": { + "hbaseThreatIntel": [ + "ip_src_addr", + "ip_dst_addr" + ] + }, + "fieldToTypeMap": { + "ip_src_addr": [ + "malicious_ip" + ], + "ip_dst_addr": [ + "malicious_ip" + ] + }, + "triageConfig" : { + "riskLevelRules" : [ + { + "rule" : "ip_src_addr == '10.0.2.3' or ip_dst_addr == '10.0.2.3'", + "score" : 10 + } + ], + "aggregator" : "MAX" + } + } +} +</pre></div></div> +<p>ThreatIntel alert levels are emitted as a new field “threat.triage.level.” So for the example above, an incoming message that trips the <tt>ip_src_addr</tt> rule will have a new field threat.triage.level=10.</p> +<p><a name="Example_Enrichment_via_Stellar"></a></p> +<h1>Example Enrichment via Stellar</h1> +<p>Let’s walk through doing a simple enrichment using Stellar on your cluster using the Squid topology.</p></div></div> +<div class="section"> +<h2><a name="Install_Prerequisites"></a>Install Prerequisites</h2> +<p>Now let’s install some prerequisites:</p> + +<ul> + +<li>Squid client via <tt>yum install squid</tt></li> + +<li>ES Head plugin via <tt>/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head</tt></li> +</ul> +<p>Start Squid via <tt>service squid start</tt></p></div> +<div class="section"> +<h2><a name="Adjust_Enrichment_Configurations_for_Squid_to_Call_Stellar"></a>Adjust Enrichment Configurations for Squid to Call Stellar</h2> +<p>Let’s adjust the configurations for the Squid topology to annotate the messages using some Stellar functions.</p> + +<ul> + +<li> +<p>Edit the squid enrichment configuration at <tt>$METRON_HOME/config/zookeeper/enrichments/squid.json</tt> (this file will not exist, so create a new one) to add some new fields based on stellar queries:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "enrichment" : { +"fieldMap": { + "stellar" : { + "config" : { + "numeric" : { + "foo": "1 + 1" + } + ,"ALL_CAPS" : "TO_UPPER(source.type)" + } + } + } + }, + "threatIntel" : { +"fieldMap":{ + "stellar" : { + "config" : { + "bar" : "TO_UPPER(source.type)" + } + } +}, +"triageConfig" : { +} + } +} +</pre></div></div> +<p>We have added the following fields as part of the enrichment phase of the enrichment topology:</p></li> + +<li> +<p><tt>foo</tt> == 2</p></li> + +<li><tt>ALL_CAPS</tt> == SQUID</li> +</ul> +<p>We have added the following as part of the threat intel:</p> + +<ul> + +<li><tt>bar</tt> == SQUID</li> +</ul> +<p>Please note that foo and ALL_CAPS will be applied in separate workers due to them being in separate groups.</p> + +<ul> + +<li>Upload new configs via <tt>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i $METRON_HOME/config/zookeeper -z node1:2181</tt></li> + +<li>Make the Squid topic in kafka via <tt>/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic squid --partitions 1 --replication-factor 1</tt></li> +</ul></div> +<div class="section"> +<h2><a name="Start_Topologies_and_Send_Data"></a>Start Topologies and Send Data</h2> +<p>Now we need to start the topologies and send some data:</p> + +<ul> + +<li>Start the squid topology via <tt>$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s squid</tt></li> + +<li>Generate some data via the squid client: + +<ul> + +<li><tt>squidclient http://yahoo.com</tt></li> + +<li><tt>squidclient http://cnn.com</tt></li> + </ul></li> + +<li>Send the data to kafka via <tt>cat /var/log/squid/access.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic squid</tt></li> + +<li>Browse the data in elasticsearch via the ES Head plugin @ <a class="externalLink" href="http://node1:9200/_plugin/head/";>http://node1:9200/_plugin/head/</a> and verify that in the squid index you have two documents</li> + +<li>Ensure that the documents have new fields <tt>foo</tt>, <tt>bar</tt> and <tt>ALL_CAPS</tt> with values as described above.</li> +</ul> +<p>Note that we could have used any Stellar statements here, including calling out to HBase via <tt>ENRICHMENT_GET</tt> and <tt>ENRICHMENT_EXISTS</tt> or even calling a machine learning model via <a href="../../metron-analytics/metron-maas-service/index.html">Model as a Service</a>.</p> +<p><a name="Notes_on_Performance_Tuning"></a></p> +<h1>Notes on Performance Tuning</h1> +<p>Default installed Metron is untuned for production deployment. There are a few knobs to tune to get the most out of your system.</p></div> +<div class="section"> +<h2><a name="Kafka_Queue"></a>Kafka Queue</h2> +<p>The <tt>enrichments</tt> kafka queue is a collection point from all of the parser topologies. As such, make sure that the number of partitions in the kafka topic is sufficient to handle the throughput that you expect from your parser topologies.</p></div> +<div class="section"> +<h2><a name="Enrichment_Topology"></a>Enrichment Topology</h2> +<p>The enrichment topology as started by the <tt>$METRON_HOME/bin/start_enrichment_topology.sh</tt> script uses a default of one executor per bolt. In a real production system, this should be customized by modifying the flux file in <tt>$METRON_HOME/flux/enrichment/remote.yaml</tt>. </p> + +<ul> + +<li>Add a <tt>parallelism</tt> field to the bolts to give Storm a parallelism hint for the various components. Give bolts which appear to be bottlenecks (e.g. stellar enrichment bolt, hbase enrichment and threat intel bolts) a larger hint.</li> + +<li>Add a <tt>parallelism</tt> field to the kafka spout which matches the number of partitions for the enrichment kafka queue.</li> + +<li>Adjust the number of workers for the topology by adjusting the <tt>topology.workers</tt> field for the topology.</li> +</ul> +<p>Finally, if workers and executors are new to you or you don’t know where to modify the flux file, the following might be of use to you:</p> + +<ul> + +<li><a class="externalLink" href="http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/";>Understanding the Parallelism of a Storm Topology</a></li> + +<li><a class="externalLink" href="http://storm.apache.org/releases/current/flux.html";>Flux Docs</a></li> +</ul></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Added: dev/incubator/metron/0.3.1-RC5-incubating/book-site/metron-platform/metron-indexing/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC5-incubating/book-site/metron-platform/metron-indexing/index.html (added) +++ dev/incubator/metron/0.3.1-RC5-incubating/book-site/metron-platform/metron-indexing/index.html Thu Feb 23 16:12:27 2017 @@ -0,0 +1,428 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-02-23 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Indexing</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> + Metron-Incubating</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Indexing</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-down"></i> + Platform</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-platform/metron-api/index.html" title="Api"> + <i class="none"></i> + Api</a> + </li> + + <li> + + <a href="../../metron-platform/metron-common/index.html" title="Common"> + <i class="none"></i> + Common</a> + </li> + + <li> + + <a href="../../metron-platform/metron-data-management/index.html" title="Data-management"> + <i class="none"></i> + Data-management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"> + <i class="none"></i> + Enrichment</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Indexing</a> + </li> + + <li> + + <a href="../../metron-platform/metron-management/index.html" title="Management"> + <i class="none"></i> + Management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> + <i class="none"></i> + Parsers</a> + </li> + + <li> + + <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"> + <i class="none"></i> + Pcap-backend</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Indexing</h1> +<p><a name="Indexing"></a></p> +<div class="section"> +<h2><a name="Introduction"></a>Introduction</h2> +<p>The <tt>indexing</tt> topology is a topology dedicated to taking the data from the enrichment topology that have been enriched and storing the data in one or more supported indices</p> + +<ul> + +<li>HDFS as rolled text files, one JSON blob per line</li> + +<li>Elasticsearch</li> + +<li>Solr</li> +</ul> +<p>By default, this topology writes out to both HDFS and one of Elasticsearch and Solr.</p> +<p>Indices are written in batch and the batch size is specified in the <a href="#Sensor_Indexing_Configuration">Sensor Indexing Configuration</a> via the <tt>batchSize</tt> parameter. This config is variable by sensor type.</p></div> +<div class="section"> +<h2><a name="Indexing_Architecture"></a>Indexing Architecture</h2> +<p><img src="../../images/indexing_arch.png" alt="Architecture" /></p> +<p>The indexing topology is extremely simple. Data is ingested into kafka and sent to </p> + +<ul> + +<li>An indexing bolt configured to write to either elasticsearch or Solr</li> + +<li>An indexing bolt configured to write to HDFS under <tt>/apps/metron/enrichment/indexed</tt></li> +</ul> +<p>Errors during indexing are sent to a kafka queue called <tt>index_errors</tt></p></div> +<div class="section"> +<h2><a name="Sensor_Indexing_Configuration"></a>Sensor Indexing Configuration</h2> +<p>The sensor specific configuration is intended to configure the indexing used for a given sensor type (e.g. <tt>snort</tt>). </p> +<p>Just like the global config, the format is a JSON stored in zookeeper and on disk at <tt>$METRON_HOME/config/zookeeper/indexing</tt>. Within the sensor-specific configuration, you can configure the individual writers. The writers currently supported are:</p> + +<ul> + +<li><tt>elasticsearch</tt></li> + +<li><tt>hdfs</tt></li> + +<li><tt>solr</tt></li> +</ul> +<p>Depending on how you start the indexing topology, it will have either elasticsearch or solr and hdfs writers running.</p> +<p>The configuration for an individual writer-specific configuration is a JSON map with the following fields:</p> + +<ul> + +<li><tt>index</tt> : The name of the index to write to (defaulted to the name of the sensor).</li> + +<li><tt>batchSize</tt> : The size of the batch that is written to the indices at once (defaulted to <tt>1</tt>).</li> + +<li><tt>enabled</tt> : Whether the writer is enabled (default <tt>true</tt>).</li> +</ul> +<div class="section"> +<h3><a name="Indexing_Configuration_Examples"></a>Indexing Configuration Examples</h3> +<p>For a given sensor, the following scenarios would be indicated by the following cases:</p> +<div class="section"> +<h4><a name="Base_Case"></a>Base Case</h4> + +<div class="source"> +<div class="source"> +<pre>{ +} +</pre></div></div> +<p>or no file at all.</p> + +<ul> + +<li>elasticsearch writer + +<ul> + +<li>enabled</li> + +<li>batch size of 1</li> + +<li>index name the same as the sensor</li> + </ul></li> + +<li>hdfs writer + +<ul> + +<li>enabled</li> + +<li>batch size of 1</li> + +<li>index name the same as the sensor</li> + </ul></li> +</ul> +<p>If a writer config is unspecified, then a warning is indicated in the Storm console. e.g.: <tt>WARNING: Default and (likely) unoptimized writer config used for hdfs writer and sensor squid</tt></p></div> +<div class="section"> +<h4><a name="Fully_specified"></a>Fully specified</h4> + +<div class="source"> +<div class="source"> +<pre>{ + "elasticsearch": { + "index": "foo", + "batchSize" : 100, + "enabled" : true + }, + "hdfs": { + "index": "foo", + "batchSize": 1, + "enabled" : true + } +} +</pre></div></div> + +<ul> + +<li>elasticsearch writer + +<ul> + +<li>enabled</li> + +<li>batch size of 100</li> + +<li>index name of “foo”</li> + </ul></li> + +<li>hdfs writer + +<ul> + +<li>enabled</li> + +<li>batch size of 1</li> + +<li>index name of “foo”</li> + </ul></li> +</ul></div> +<div class="section"> +<h4><a name="HDFS_Writer_turned_off"></a>HDFS Writer turned off</h4> + +<div class="source"> +<div class="source"> +<pre>{ + "elasticsearch": { + "index": "foo", + "enabled" : true + }, + "hdfs": { + "index": "foo", + "batchSize": 100, + "enabled" : false + } +} +</pre></div></div> + +<ul> + +<li>elasticsearch writer + +<ul> + +<li>enabled</li> + +<li>batch size of 1</li> + +<li>index name of “foo”</li> + </ul></li> + +<li>hdfs writer + +<ul> + +<li>disabled</li> + </ul></li> +</ul> +<p><a name="Notes_on_Performance_Tuning"></a></p> +<h1>Notes on Performance Tuning</h1> +<p>Default installed Metron is untuned for production deployment. By far and wide, the most likely piece to require TLC from a performance perspective is the indexing layer. An index that does not keep up will back up and you will see errors in the kafka bolt. There are a few knobs to tune to get the most out of your system.</p></div></div></div> +<div class="section"> +<h2><a name="Kafka_Queue"></a>Kafka Queue</h2> +<p>The <tt>indexing</tt> kafka queue is a collection point from the enrichment topology. As such, make sure that the number of partitions in the kafka topic is sufficient to handle the throughput that you expect.</p></div> +<div class="section"> +<h2><a name="Indexing_Topology"></a>Indexing Topology</h2> +<p>The enrichment topology as started by the <tt>$METRON_HOME/bin/start_elasticsearch_topology.sh</tt> or <tt>$METRON_HOME/bin/start_solr_topology.sh</tt> script uses a default of one executor per bolt. In a real production system, this should be customized by modifying the flux file in <tt>$METRON_HOME/flux/indexing/remote.yaml</tt>. </p> + +<ul> + +<li>Add a <tt>parallelism</tt> field to the bolts to give Storm a parallelism hint for the various components. Give bolts which appear to be bottlenecks (e.g. the indexing bolt) a larger hint.</li> + +<li>Add a <tt>parallelism</tt> field to the kafka spout which matches the number of partitions for the enrichment kafka queue.</li> + +<li>Adjust the number of workers for the topology by adjusting the <tt>topology.workers</tt> field for the topology.</li> +</ul> +<p>Finally, if workers and executors are new to you or you don’t know where to modify the flux file, the following might be of use to you:</p> + +<ul> + +<li><a class="externalLink" href="http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/";>Understanding the Parallelism of a Storm Topology</a></li> + +<li><a class="externalLink" href="http://storm.apache.org/releases/current/flux.html";>Flux Docs</a></li> +</ul></div> +<div class="section"> +<h2><a name="Zeppelin_Notebooks"></a>Zeppelin Notebooks</h2> +<p>Zeppelin notebooks can be added to <tt>/src/main/config/zeppelin/</tt> (and subdirectories can be created for organization). The placed files must be .json files and be named appropriately. These files must be added to the metron.spec file and the RPMs rebuilt to be available to be loaded into Ambari.</p> +<p>The notebook files will be found on the server in <tt>$METRON_HOME/config/zeppelin</tt></p> +<p>The Ambari Management Pack has a custom action to load these templates, ZEPPELIN_DASHBOARD_INSTALL, that will import them into Zeppelin.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
