http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-platform/metron-enrichment/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-enrichment/index.html
b/site/current-book/metron-platform/metron-enrichment/index.html
index 584c36f..de1e4fa 100644
--- a/site/current-book/metron-platform/metron-enrichment/index.html
+++ b/site/current-book/metron-platform/metron-enrichment/index.html
@@ -1,291 +1,165 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2018-01-03
- | Rendered using Apache Maven Fluido Skin 1.3.0
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from
src/site/markdown/metron-platform/metron-enrichment/index.md at 2018-06-07
+ | Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180103" />
+ <meta name="Date-Revision-yyyymmdd" content="20180607" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Enrichment</title>
- <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css"
/>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
<link rel="stylesheet" href="../../css/site.css" />
<link rel="stylesheet" href="../../css/print.css" media="print" />
-
-
- <script type="text/javascript"
src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
-
-
-
-<script type="text/javascript">$( document ).ready( function() { $(
'.carousel' ).carousel( { interval: 3500 } ) } );</script>
-
- </head>
- <body class="topBarDisabled">
-
-
-
-
- <div class="container-fluid">
- <div id="banner">
- <div class="pull-left">
- <a href="http://metron.apache.org/";
id="bannerLeft">
-
<img src="../../images/metron-logo.png" alt="Apache Metron"
width="148px" height="48px"/>
- </a>
- </div>
- <div class="pull-right"> </div>
+ <script type="text/javascript"
src="../../js/apache-maven-fluido-1.7.min.js"></script>
+<script type="text/javascript">
+ $( document ).ready( function() { $( '.carousel' ).carousel( {
interval: 3500 } ) } );
+ </script>
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href="http://metron.apache.org/";
id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron"
width="148px" height="48px"/></a></div>
+ <div class="pull-right"></div>
<div class="clear"><hr/></div>
</div>
<div id="breadcrumbs">
<ul class="breadcrumb">
-
-
- <li class="">
- <a href="http://www.apache.org"; class="externalLink"
title="Apache">
- Apache</a>
- </li>
- <li class="divider ">/</li>
- <li class="">
- <a href="http://metron.apache.org/"; class="externalLink"
title="Metron">
- Metron</a>
- </li>
- <li class="divider ">/</li>
- <li class="">
- <a href="../../index.html" title="Documentation">
- Documentation</a>
- </li>
- <li class="divider ">/</li>
- <li class="">Enrichment</li>
-
-
-
- <li id="publishDate" class="pull-right">Last Published:
2018-01-03</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
-
- </ul>
+ <li class=""><a href="http://www.apache.org"; class="externalLink"
title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="http://metron.apache.org/"; class="externalLink"
title="Metron">Metron</a><span class="divider">/</span></li>
+ <li class=""><a href="../../index.html"
title="Documentation">Documentation</a><span class="divider">/</span></li>
+ <li class="active ">Enrichment</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span>
Last Published: 2018-06-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.5.0</li>
+ </ul>
</div>
-
-
<div class="row-fluid">
- <div id="leftColumn" class="span3">
+ <div id="leftColumn" class="span2">
<div class="well sidebar-nav">
-
-
- <ul class="nav nav-list">
- <li class="nav-header">User Documentation</li>
-
- <li>
-
- <a href="../../index.html" title="Metron">
- <i class="icon-chevron-down"></i>
- Metron</a>
- <ul class="nav nav-list">
-
- <li>
-
- <a href="../../Upgrading.html" title="Upgrading">
- <i class="none"></i>
- Upgrading</a>
- </li>
-
- <li>
-
- <a href="../../metron-analytics/index.html"
title="Analytics">
- <i class="icon-chevron-right"></i>
- Analytics</a>
- </li>
-
- <li>
-
- <a
href="../../metron-contrib/metron-docker/index.html" title="Docker">
- <i class="none"></i>
- Docker</a>
- </li>
-
- <li>
-
- <a href="../../metron-deployment/index.html"
title="Deployment">
- <i class="icon-chevron-right"></i>
- Deployment</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-alerts/index.html" title="Alerts">
- <i class="none"></i>
- Alerts</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-config/index.html" title="Config">
- <i class="none"></i>
- Config</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-rest/index.html" title="Rest">
- <i class="none"></i>
- Rest</a>
- </li>
-
- <li>
-
- <a href="../../metron-platform/index.html"
title="Platform">
- <i class="icon-chevron-down"></i>
- Platform</a>
- <ul class="nav nav-list">
-
- <li>
-
- <a
href="../../metron-platform/Performance-tuning-guide.html"
title="Performance-tuning-guide">
- <i class="none"></i>
- Performance-tuning-guide</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-api/index.html" title="Api">
- <i class="none"></i>
- Api</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-common/index.html" title="Common">
- <i class="none"></i>
- Common</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-data-management/index.html"
title="Data-management">
- <i class="none"></i>
- Data-management</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-elasticsearch/index.html"
title="Elasticsearch">
- <i class="none"></i>
- Elasticsearch</a>
- </li>
-
- <li class="active">
-
- <a href="#"><i class="none"></i>Enrichment</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-indexing/index.html" title="Indexing">
- <i class="none"></i>
- Indexing</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-management/index.html" title="Management">
- <i class="none"></i>
- Management</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-parsers/index.html" title="Parsers">
- <i class="icon-chevron-right"></i>
- Parsers</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-pcap-backend/index.html"
title="Pcap-backend">
- <i class="none"></i>
- Pcap-backend</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-writer/index.html" title="Writer">
- <i class="none"></i>
- Writer</a>
- </li>
- </ul>
- </li>
-
- <li>
-
- <a href="../../metron-sensors/index.html"
title="Sensors">
- <i class="icon-chevron-right"></i>
- Sensors</a>
- </li>
-
- <li>
-
- <a
href="../../metron-stellar/stellar-3rd-party-example/index.html"
title="Stellar-3rd-party-example">
- <i class="none"></i>
- Stellar-3rd-party-example</a>
- </li>
-
- <li>
-
- <a
href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
- <i class="icon-chevron-right"></i>
- Stellar-common</a>
- </li>
-
- <li>
-
- <a href="../../use-cases/index.html"
title="Use-cases">
- <i class="icon-chevron-right"></i>
- Use-cases</a>
- </li>
- </ul>
- </li>
- </ul>
-
-
-
- <hr class="divider" />
-
- <div id="poweredBy">
- <div class="clear"></div>
- <div class="clear"></div>
- <div class="clear"></div>
- <a href="http://maven.apache.org/"; title="Built
by Maven" class="poweredBy">
- <img class="builtBy" alt="Built by Maven"
src="../../images/logos/maven-feather.png" />
- </a>
- </div>
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+ <li><a href="../../index.html" title="Metron"><span
class="icon-chevron-down"></span>Metron</a>
+ <ul class="nav nav-list">
+ <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span
class="none"></span>CONTRIBUTING</a></li>
+ <li><a href="../../Upgrading.html" title="Upgrading"><span
class="none"></span>Upgrading</a></li>
+ <li><a href="../../metron-analytics/index.html" title="Analytics"><span
class="icon-chevron-right"></span>Analytics</a></li>
+ <li><a href="../../metron-contrib/metron-docker/index.html"
title="Docker"><span class="none"></span>Docker</a></li>
+ <li><a href="../../metron-contrib/metron-performance/index.html"
title="Performance"><span class="none"></span>Performance</a></li>
+ <li><a href="../../metron-deployment/index.html" title="Deployment"><span
class="icon-chevron-right"></span>Deployment</a></li>
+ <li><a href="../../metron-interface/metron-alerts/index.html"
title="Alerts"><span class="none"></span>Alerts</a></li>
+ <li><a href="../../metron-interface/metron-config/index.html"
title="Config"><span class="none"></span>Config</a></li>
+ <li><a href="../../metron-interface/metron-rest/index.html"
title="Rest"><span class="none"></span>Rest</a></li>
+ <li><a href="../../metron-platform/index.html" title="Platform"><span
class="icon-chevron-down"></span>Platform</a>
+ <ul class="nav nav-list">
+ <li><a href="../../metron-platform/Performance-tuning-guide.html"
title="Performance-tuning-guide"><span
class="none"></span>Performance-tuning-guide</a></li>
+ <li><a href="../../metron-platform/metron-api/index.html"
title="Api"><span class="none"></span>Api</a></li>
+ <li><a href="../../metron-platform/metron-common/index.html"
title="Common"><span class="none"></span>Common</a></li>
+ <li><a href="../../metron-platform/metron-data-management/index.html"
title="Data-management"><span class="none"></span>Data-management</a></li>
+ <li><a href="../../metron-platform/metron-elasticsearch/index.html"
title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li>
+ <li class="active"><a href="#"><span
class="icon-chevron-down"></span>Enrichment</a>
+ <ul class="nav nav-list">
+ <li><a href="../../metron-platform/metron-enrichment/Performance.html"
title="Performance"><span class="none"></span>Performance</a></li>
+ </ul>
+</li>
+ <li><a href="../../metron-platform/metron-indexing/index.html"
title="Indexing"><span class="none"></span>Indexing</a></li>
+ <li><a href="../../metron-platform/metron-management/index.html"
title="Management"><span class="none"></span>Management</a></li>
+ <li><a href="../../metron-platform/metron-parsers/index.html"
title="Parsers"><span class="icon-chevron-right"></span>Parsers</a></li>
+ <li><a href="../../metron-platform/metron-pcap-backend/index.html"
title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li>
+ <li><a href="../../metron-platform/metron-writer/index.html"
title="Writer"><span class="none"></span>Writer</a></li>
+ </ul>
+</li>
+ <li><a href="../../metron-sensors/index.html" title="Sensors"><span
class="icon-chevron-right"></span>Sensors</a></li>
+ <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html"
title="Stellar-3rd-party-example"><span
class="none"></span>Stellar-3rd-party-example</a></li>
+ <li><a href="../../metron-stellar/stellar-common/index.html"
title="Stellar-common"><span
class="icon-chevron-right"></span>Stellar-common</a></li>
+ <li><a href="../../metron-stellar/stellar-zeppelin/index.html"
title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
+ <li><a href="../../use-cases/index.html" title="Use-cases"><span
class="icon-chevron-right"></span>Use-cases</a></li>
+ </ul>
+</li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href="http://maven.apache.org/"; title="Built by Maven"
class="poweredBy"><img class="builtBy" alt="Built by Maven"
src="../../images/logos/maven-feather.png" /></a>
+ </div>
</div>
</div>
-
-
- <div id="bodyColumn" class="span9" >
-
- <h1>Enrichment</h1>
+ <div id="bodyColumn" class="span10" >
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<h1>Enrichment</h1>
<p><a name="Enrichment"></a></p>
<div class="section">
<h2><a name="Introduction"></a>Introduction</h2>
-<p>The <tt>enrichment</tt> topology is a topology dedicated to taking the data
from the parsing topologies that have been normalized into the Metron data
format (e.g. a JSON Map structure with <tt>original_message</tt> and
<tt>timestamp</tt>) and </p>
-
+<p>The <tt>enrichment</tt> topology is a topology dedicated to taking the data
from the parsing topologies that have been normalized into the Metron data
format (e.g. a JSON Map structure with <tt>original_message</tt> and
<tt>timestamp</tt>) and</p>
<ul>
-
-<li>Enriching messages with external data from data stores (e.g. hbase) by
adding new fields based on existing fields in the messages.</li>
-
+
+<li>Enriching messages with external data from data stores (e.g. hbase) by
adding new fields based on existing fields in the messages.</li>
<li>Marking messages as threats based on data in external data stores</li>
-
-<li>Marking threat alerts with a numeric triage level based on a set of
Stellar rules.</li>
+<li>Marking threat alerts with a numeric triage level based on a set of
Stellar rules.</li>
</ul></div>
<div class="section">
<h2><a name="Enrichment_Architecture"></a>Enrichment Architecture</h2>
-<p><img src="../../images/enrichment_arch.png" alt="Architecture" /></p></div>
+<p><img src="../../images/enrichment_arch.png" alt="Architecture" /></p>
+<div class="section">
+<h3><a name="Unified_Enrichment_Topology"></a>Unified Enrichment Topology</h3>
+<p>There is an experimental unified enrichment topology which is shipped.
Currently the architecture, as described above, has a split/join in order to
perform enrichments in parallel. This poses some issues in terms of ease of
tuning and reasoning about performance.</p>
+<p>In order to deal with these issues, there is an alternative enrichment
topology which uses data parallelism as opposed to the split/join task
parallelism. This architecture uses a worker pool to fully enrich any message
within a worker. This results in</p>
+<ul>
+
+<li>Fewer bolts in the topology</li>
+<li>Each bolt fully operates on a message.</li>
+<li>Fewer network hops</li>
+</ul>
+<p><img src="unified_enrichment_arch.svg" alt="Unified Architecture" /></p>
+<p>This architecture is fully backwards compatible; the only difference is how
the enrichment will operate on each message (in one bolt where the split/join
is done in a threadpool as opposed to split across multiple bolts).</p>
+<div class="section">
+<h4><a name="Using_It"></a>Using It</h4>
+<p>In order to use this, you will need to</p>
+<ul>
+
+<li>Edit <tt>$METRON_HOME/bin/start_enrichment_topology.sh</tt> and adjust it
to use <tt>remote-unified.yaml</tt> instead of <tt>remote.yaml</tt></li>
+<li>Restart the enrichment topology.</li>
+</ul></div>
+<div class="section">
+<h4><a name="Configuring_It"></a>Configuring It</h4>
+<p>There are two parameters which you might want to tune in this topology.
Both of them are topology configuration adjustable in the flux file
<tt>$METRON_HOME/config/flux/enrichment/remote-unified.yaml</tt>:</p>
+<ul>
+
+<li><tt>metron.threadpool.size</tt> : The size of the threadpool. This can
take a number or a multiple of the number of cores (e.g. <tt>5C</tt> to 5 times
the number of cores). The default is <tt>2C</tt>.</li>
+<li><tt>metron.threadpool.type</tt> : The type of threadpool. (note:
descriptions taken from <a class="externalLink"
href="https://zeroturnaround.com/rebellabs/fixedthreadpool-cachedthreadpool-or-forkjoinpool-picking-correct-java-executors-for-background-tasks/";>here</a>).
+<ul>
+
+<li><tt>FIXED</tt> is a fixed threadpool of size <tt>n</tt>. <tt>n</tt>
threads will process tasks at the time, when the pool is saturated, new tasks
will get added to a queue without a limit on size. Good for CPU intensive
tasks. This is the default.</li>
+<li><tt>WORK_STEALING</tt> is a work stealing threadpool. This will create
and shut down threads dynamically to accommodate the required parallelism
level. It also tries to reduce the contention on the task queue, so can be
really good in heavily loaded environments. Also good when your tasks create
more tasks for the executor, like recursive tasks.</li>
+</ul>
+</li>
+</ul>
+<p>In order to configure the parallelism for the enrichment bolt and threat
intel bolt, the configurations will be taken from the respective join bolt
parallelism. When proper ambari support for this is added, we will add its own
property.</p></div></div></div>
<div class="section">
<h2><a name="Enrichment_Configuration"></a>Enrichment Configuration</h2>
<p>The configuration for the <tt>enrichment</tt> topology, the topology
primarily responsible for enrichment and threat intelligence enrichment, is
defined by JSON documents stored in zookeeper.</p>
-<p>There are two types of configurations at the moment, <tt>global</tt> and
<tt>sensor</tt> specific. </p></div>
+<p>There are two types of configurations at the moment, <tt>global</tt> and
<tt>sensor</tt> specific.</p></div>
<div class="section">
<h2><a name="Global_Configuration"></a>Global Configuration</h2>
<p>There are a few enrichments which have independent configurations, such as
from the global config.</p>
@@ -295,69 +169,49 @@
<p>Metron supports enrichment of IP information using <a class="externalLink"
href="https://dev.maxmind.com/geoip/geoip2/geolite2/";>GeoLite2</a>. The
location of the file is managed in the global config.</p>
<div class="section">
<h4><a name="geo.hdfs.file"></a><tt>geo.hdfs.file</tt></h4>
-<p>The location on HDFS of the GeoLite2 database file to use for GeoIP
lookups. This file will be localized on the storm supervisors running the
topology and used from there. This is lazy, so if this property changes in a
running topology, the file will be localized from HDFS upon first time the file
is used via the geo enrichment. </p></div></div></div>
+<p>The location on HDFS of the GeoLite2 database file to use for GeoIP
lookups. This file will be localized on the storm supervisors running the
topology and used from there. This is lazy, so if this property changes in a
running topology, the file will be localized from HDFS upon first time the file
is used via the geo enrichment.</p></div></div></div>
<div class="section">
<h2><a name="Sensor_Enrichment_Configuration"></a>Sensor Enrichment
Configuration</h2>
<p>The sensor specific configuration is intended to configure the individual
enrichments and threat intelligence enrichments for a given sensor type (e.g.
<tt>snort</tt>).</p>
<p>Just like the global config, the format is a JSON stored in zookeeper. The
configuration is a complex JSON object with the following top level fields:</p>
-
<ul>
-
+
<li><tt>enrichment</tt> : A complex JSON object representing the configuration
of the enrichments</li>
-
<li><tt>threatIntel</tt> : A complex JSON object representing the
configuration of the threat intelligence enrichments</li>
</ul>
<div class="section">
<h3><a name="The_enrichment_Configuration"></a>The <tt>enrichment</tt>
Configuration</h3>
-
<table border="0" class="table table-striped">
- <thead>
-
+<thead>
+
<tr class="a">
-
-<th>Field </th>
-
-<th>Description </th>
-
-<th>Example </th>
- </tr>
- </thead>
- <tbody>
-
+<th> Field </th>
+<th> Description
</th>
+<th> Example
</th></tr>
+</thead><tbody>
+
<tr class="b">
-
-<td><tt>fieldToTypeMap</tt> </td>
-
-<td>In the case of a simple HBase enrichment (i.e. a key/value lookup), the
mapping between fields and the enrichment types associated with those fields
must be known. This enrichment type is used as part of the HBase key. Note:
applies to hbaseEnrichment only. </td>
-
-<td><tt>"fieldToTypeMap" : { "ip_src_addr" : [
"asset_enrichment" ] }</tt> </td>
- </tr>
-
+<td> <tt>fieldToTypeMap</tt> </td>
+<td> In the case of a simple HBase enrichment (i.e. a key/value lookup), the
mapping between fields and the enrichment types associated with those fields
must be known. This enrichment type is used as part of the HBase key. Note:
applies to hbaseEnrichment only. </td>
+<td> <tt>"fieldToTypeMap" : { "ip_src_addr" : [
"asset_enrichment" ] }</tt> </td></tr>
<tr class="a">
-
-<td><tt>fieldMap</tt> </td>
-
-<td>The map of enrichment bolts names to configuration handlers which know how
to split the message up. The simplest of which is just a list of fields. More
complex examples would be the stellar enrichment which provides stellar
statements. Each field listed in the array arg is sent to the enrichment
referenced in the key. Cardinality of fields to enrichments is many-to-many.
</td>
-
-<td><tt>"fieldMap": {"hbaseEnrichment":
["ip_src_addr","ip_dst_addr"]}</tt> </td>
- </tr>
-
+<td> <tt>fieldMap</tt> </td>
+<td> The map of enrichment bolts names to configuration handlers which know
how to split the message up. The simplest of which is just a list of fields.
More complex examples would be the stellar enrichment which provides stellar
statements. Each field listed in the array arg is sent to the enrichment
referenced in the key. Cardinality of fields to enrichments is many-to-many.
</td>
+<td> <tt>"fieldMap": {"hbaseEnrichment":
["ip_src_addr","ip_dst_addr"]}</tt> </td></tr>
<tr class="b">
-
-<td><tt>config</tt> </td>
-
-<td>The general configuration for the enrichment </td>
-
-<td><tt>"config": {"typeToColumnFamily": {
"asset_enrichment" : "cf" } }</tt> </td>
- </tr>
- </tbody>
+<td> <tt>config</tt> </td>
+<td> The general configuration for the enrichment
</td>
+<td> <tt>"config": {"typeToColumnFamily": {
"asset_enrichment" : "cf" } }</tt> </td></tr>
+</tbody>
</table>
<p>The <tt>config</tt> map is intended to house enrichment specific
configuration. For instance, for the <tt>hbaseEnrichment</tt>, the mappings
between the enrichment types to the column families is specified.</p>
-<p>The <tt>fieldMap</tt>contents are of interest because they contain the
routing and configuration information for the enrichments.<br />When we say
‘routing’, we mean how the messages get split up and sent to the
enrichment adapter bolts.<br />The simplest, by far, is just providing a simple
list as in</p>
+<p>The <tt>fieldMap</tt>contents are of interest because they contain the
routing and configuration information for the enrichments.<br />
+When we say ‘routing’, we mean how the messages get split up and
sent to the enrichment adapter bolts.<br />
+The simplest, by far, is just providing a simple list as in</p>
-<div class="source">
-<div class="source">
-<pre> "fieldMap": {
+<div>
+<div>
+<pre class="source"> "fieldMap": {
"geo": [
"ip_src_addr",
"ip_dst_addr"
@@ -372,15 +226,16 @@
]
}
</pre></div></div>
-<p>Based on this sample config, both <tt>ip_src_addr</tt> and
<tt>ip_dst_addr</tt> will go to the <tt>geo</tt>, <tt>host</tt>, and
<tt>hbaseEnrichment</tt> adapter bolts. </p>
+
+<p>Based on this sample config, both <tt>ip_src_addr</tt> and
<tt>ip_dst_addr</tt> will go to the <tt>geo</tt>, <tt>host</tt>, and
<tt>hbaseEnrichment</tt> adapter bolts.</p>
<div class="section">
<h4><a name="Stellar_Enrichment_Configuration"></a>Stellar Enrichment
Configuration</h4>
-<p>For the <tt>geo</tt>, <tt>host</tt> and <tt>hbaseEnrichment</tt>, this is
sufficient. However, more complex enrichments may contain their own
configuration. Currently, the <tt>stellar</tt> enrichment is more adaptable and
thus requires a more nuanced configuration.</p>
-<p>At its most basic, we want to take a message and apply a couple of
enrichments, such as converting the <tt>hostname</tt> field to lowercase. We do
this by specifying the transformation inside of the <tt>config</tt> for the
<tt>stellar</tt> fieldMap. There are two syntaxes that are supported,
specifying the transformations as a map with the key as the field and the value
the stellar expression:</p>
+<p>For the <tt>geo</tt>, <tt>host</tt> and <tt>hbaseEnrichment</tt>, this is
sufficient. However, more complex enrichments may contain their own
configuration. Currently, the <tt>stellar</tt> enrichment is more adaptable
and thus requires a more nuanced configuration.</p>
+<p>At its most basic, we want to take a message and apply a couple of
enrichments, such as converting the <tt>hostname</tt> field to lowercase. We do
this by specifying the transformation inside of the <tt>config</tt> for the
<tt>stellar</tt> fieldMap. There are two syntaxes that are supported,
specifying the transformations as a map with the key as the field and the value
the stellar expression:</p>
-<div class="source">
-<div class="source">
-<pre> "fieldMap": {
+<div>
+<div>
+<pre class="source"> "fieldMap": {
...
"stellar" : {
"config" : {
@@ -389,11 +244,12 @@
}
}
</pre></div></div>
+
<p>Another approach is to make the transformations as a list with the same
<tt>var := expr</tt> syntax as is used in the Stellar REPL, such as:</p>
-<div class="source">
-<div class="source">
-<pre> "fieldMap": {
+<div>
+<div>
+<pre class="source"> "fieldMap": {
...
"stellar" : {
"config" : [
@@ -402,12 +258,13 @@
}
}
</pre></div></div>
-<p>Sometimes arbitrary stellar enrichments may take enough time that you would
prefer to split some of them into groups and execute the groups of stellar
enrichments in parallel. Take, for instance, if you wanted to do an HBase
enrichment and a profiler call which were independent of one another. This
usecase is supported by splitting the enrichments up as groups.</p>
+
+<p>Sometimes arbitrary stellar enrichments may take enough time that you would
prefer to split some of them into groups and execute the groups of stellar
enrichments in parallel. Take, for instance, if you wanted to do an HBase
enrichment and a profiler call which were independent of one another. This
usecase is supported by splitting the enrichments up as groups.</p>
<p>Consider the following example:</p>
-<div class="source">
-<div class="source">
-<pre> "fieldMap": {
+<div>
+<div>
+<pre class="source"> "fieldMap": {
...
"stellar" : {
"config" : {
@@ -428,143 +285,93 @@
}
}
</pre></div></div>
-<p>Here we want to perform two enrichments that hit HBase and we would rather
not run in sequence. These enrichments are entirely independent of one another
(i.e. neither relies on the output of the other). In this case, we’ve
created a group called <tt>malicious_domain_enrichment</tt> to inquire about
whether the destination address exists in the HBase enrichment table in the
<tt>malicious_domains</tt> enrichment type. This is a simple enrichment, so we
can express the enrichment group as a map with the new field
<tt>is_bad_domain</tt> being a key and the stellar expression associated with
that operation being the associated value.</p>
-<p>In contrast, the stellar enrichment group <tt>login_profile</tt> is
interacting with the profiler, has multiple temporary expressions (i.e.
<tt>profile_window</tt>, <tt>global_login_profile</tt>, and <tt>stats</tt>)
that are useful only within the context of this group of stellar expressions.
In this case, we would need to ensure that we use the list construct when
specifying the group and remember to set the temporary variables to
<tt>null</tt> so they are not passed along.</p>
-<p>In general, things to note from this section are as follows:</p>
+<p>Here we want to perform two enrichments that hit HBase and we would rather
not run in sequence. These enrichments are entirely independent of one another
(i.e. neither relies on the output of the other). In this case, we’ve
created a group called <tt>malicious_domain_enrichment</tt> to inquire about
whether the destination address exists in the HBase enrichment table in the
<tt>malicious_domains</tt> enrichment type. This is a simple enrichment, so we
can express the enrichment group as a map with the new field
<tt>is_bad_domain</tt> being a key and the stellar expression associated with
that operation being the associated value.</p>
+<p>In contrast, the stellar enrichment group <tt>login_profile</tt> is
interacting with the profiler, has multiple temporary expressions (i.e.
<tt>profile_window</tt>, <tt>global_login_profile</tt>, and <tt>stats</tt>)
that are useful only within the context of this group of stellar expressions.
In this case, we would need to ensure that we use the list construct when
specifying the group and remember to set the temporary variables to
<tt>null</tt> so they are not passed along.</p>
+<p>In general, things to note from this section are as follows:</p>
<ul>
-
+
<li>The stellar enrichments for the <tt>stellar</tt> enrichment adapter are
specified in the <tt>config</tt> for the <tt>stellar</tt> enrichment adapter in
the <tt>fieldMap</tt></li>
-
<li>Groups of independent (i.e. no expression in any group depend on the
output of an expression from an other group) may be executed in parallel</li>
-
-<li>If you have the need to use temporary variables, you may use the list
construct. Ensure that you assign the variables to <tt>null</tt> before the end
of the group.</li>
-
+<li>If you have the need to use temporary variables, you may use the list
construct. Ensure that you assign the variables to <tt>null</tt> before the
end of the group.</li>
<li><b>Ensure that you do not assign a field to a stellar expression which
returns an object which JSON cannot represent.</b></li>
-
<li>Fields assigned to Maps as part of stellar enrichments have the maps
unfolded, similar to the HBase Enrichment
-
<ul>
-
-<li>For example the stellar enrichment for field <tt>foo</tt> which assigns a
map such as <tt>foo := { 'grok' : 1, 'bar' : 'baz'}</tt> would yield the
following fields:
-
+
+<li>For example the stellar enrichment for field <tt>foo</tt> which assigns a
map such as <tt>foo := { 'grok' : 1, 'bar' : 'baz'}</tt> would yield the
following fields:
<ul>
-
+
<li><tt>foo.grok</tt> == <tt>1</tt></li>
-
<li><tt>foo.bar</tt> == <tt>'baz'</tt></li>
- </ul></li>
- </ul></li>
+</ul>
+</li>
+</ul>
+</li>
</ul></div></div>
<div class="section">
<h3><a name="The_threatIntel_Configuration"></a>The <tt>threatIntel</tt>
Configuration</h3>
-
<table border="0" class="table table-striped">
- <thead>
-
+<thead>
+
<tr class="a">
-
-<th>Field </th>
-
-<th>Description </th>
-
-<th>Example </th>
- </tr>
- </thead>
- <tbody>
-
+<th> Field </th>
+<th> Description
</th>
+<th> Example
</th></tr>
+</thead><tbody>
+
<tr class="b">
-
-<td><tt>fieldToTypeMap</tt> </td>
-
-<td>In the case of a simple HBase threat intel enrichment (i.e. a key/value
lookup), the mapping between fields and the enrichment types associated with
those fields must be known. This enrichment type is used as part of the HBase
key. Note: applies to hbaseThreatIntel only. </td>
-
-<td><tt>"fieldToTypeMap" : { "ip_src_addr" : [
"malicious_ips" ] }</tt> </td>
- </tr>
-
+<td> <tt>fieldToTypeMap</tt> </td>
+<td> In the case of a simple HBase threat intel enrichment (i.e. a key/value
lookup), the mapping between fields and the enrichment types associated with
those fields must be known. This enrichment type is used as part of the HBase
key. Note: applies to hbaseThreatIntel only. </td>
+<td> <tt>"fieldToTypeMap" : { "ip_src_addr" : [
"malicious_ips" ] }</tt> </td></tr>
<tr class="a">
-
-<td><tt>fieldMap</tt> </td>
-
-<td>The map of threat intel enrichment bolts names to fields in the JSON
messages. Each field is sent to the threat intel enrichment bolt referenced in
the key. Each field listed in the array arg is sent to the enrichment
referenced in the key. Cardinality of fields to enrichments is many-to-many.
</td>
-
-<td><tt>"fieldMap": {"hbaseThreatIntel":
["ip_src_addr","ip_dst_addr"]}</tt> </td>
- </tr>
-
+<td> <tt>fieldMap</tt> </td>
+<td> The map of threat intel enrichment bolts names to fields in the JSON
messages. Each field is sent to the threat intel enrichment bolt referenced in
the key. Each field listed in the array arg is sent to the enrichment
referenced in the key. Cardinality of fields to enrichments is many-to-many.
</td>
+<td> <tt>"fieldMap": {"hbaseThreatIntel":
["ip_src_addr","ip_dst_addr"]}</tt> </td></tr>
<tr class="b">
-
-<td><tt>triageConfig</tt> </td>
-
-<td>The configuration of the threat triage scorer. In the situation where a
threat is detected, a score is assigned to the message and embedded in the
indexed message. </td>
-
-<td><tt>"riskLevelRules" : { "IN_SUBNET(ip_dst_addr,
'192.168.0.0/24')" : 10 }</tt> </td>
- </tr>
-
+<td> <tt>triageConfig</tt> </td>
+<td> The configuration of the threat triage scorer. In the situation where a
threat is detected, a score is assigned to the message and embedded in the
indexed message.
</td>
+<td> <tt>"riskLevelRules" : { "IN_SUBNET(ip_dst_addr,
'192.168.0.0/24')" : 10 }</tt> </td></tr>
<tr class="a">
-
-<td><tt>config</tt> </td>
-
-<td>The general configuration for the Threat Intel </td>
-
-<td><tt>"config": {"typeToColumnFamily": {
"malicious_ips","cf" } }</tt> </td>
- </tr>
- </tbody>
+<td> <tt>config</tt> </td>
+<td> The general configuration for the Threat Intel
</td>
+<td> <tt>"config": {"typeToColumnFamily": {
"malicious_ips","cf" } }</tt> </td></tr>
+</tbody>
</table>
-<p>The <tt>config</tt> map is intended to house threat intel specific
configuration. For instance, for the <tt>hbaseThreatIntel</tt> threat intel
adapter, the mappings between the enrichment types to the column families is
specified. The <tt>fieldMap</tt> configuration is similar to the
<tt>enrichment</tt> configuration in that the adapters available are the
same.</p>
+<p>The <tt>config</tt> map is intended to house threat intel specific
configuration. For instance, for the <tt>hbaseThreatIntel</tt> threat intel
adapter, the mappings between the enrichment types to the column families is
specified. The <tt>fieldMap</tt> configuration is similar to the
<tt>enrichment</tt> configuration in that the adapters available are the
same.</p>
<p>The <tt>triageConfig</tt> field is also a complex field and it bears some
description:</p>
-
<table border="0" class="table table-striped">
- <thead>
-
+<thead>
+
<tr class="a">
-
-<th>Field </th>
-
-<th>Description </th>
-
-<th>Example </th>
- </tr>
- </thead>
- <tbody>
-
+<th> Field </th>
+<th> Description
</th>
+<th> Example
</th></tr>
+</thead><tbody>
+
<tr class="b">
-
-<td><tt>riskLevelRules</tt> </td>
-
-<td>This is a list of rules (represented as Stellar expressions) associated
with scores with optional names and comments </td>
-
-<td>see below</td>
- </tr>
-
+<td> <tt>riskLevelRules</tt> </td>
+<td> This is a list of rules (represented as Stellar expressions) associated
with scores with optional names and comments
</td>
+<td> see below</td></tr>
<tr class="a">
-
-<td><tt>aggregator</tt> </td>
-
-<td>An aggregation function that takes all non-zero scores representing the
matching queries from <tt>riskLevelRules</tt> and aggregates them into a single
score. </td>
-
-<td><tt>"MAX"</tt> </td>
- </tr>
- </tbody>
+<td> <tt>aggregator</tt> </td>
+<td> An aggregation function that takes all non-zero scores representing the
matching queries from <tt>riskLevelRules</tt> and aggregates them into a single
score. </td>
+<td> <tt>"MAX"</tt>
</td></tr>
+</tbody>
</table>
<p>A risk level rule is of the following format:</p>
-
<ul>
-
+
<li><tt>name</tt> : The name of the threat triage rule</li>
-
<li><tt>comment</tt> : A comment describing the rule</li>
-
<li><tt>rule</tt> : The rule, represented as a Stellar statement</li>
-
<li><tt>score</tt> : Associated threat triage score for the rule</li>
-
<li><tt>reason</tt> : Reason the rule tripped. Can be represented as a Stellar
statement</li>
</ul>
<p>An example of a rule is as follows:</p>
-<div class="source">
-<div class="source">
-<pre> "riskLevelRules" : [
+<div>
+<div>
+<pre class="source"> "riskLevelRules" : [
{
"name" : "is internal"
, "comment" : "determines if the destination is
internal."
@@ -574,27 +381,23 @@
}
]
</pre></div></div>
-<p>The supported aggregation functions are:</p>
+<p>The supported aggregation functions are:</p>
<ul>
-
+
<li><tt>MAX</tt> : The max of all of the associated values for matching
queries</li>
-
<li><tt>MIN</tt> : The min of all of the associated values for matching
queries</li>
-
<li><tt>MEAN</tt> : The mean of all of the associated values for matching
queries</li>
-
<li><tt>SUM</tt> : The sum of all the associated values for matching
queries</li>
-
<li><tt>POSITIVE_MEAN</tt> : The mean of the positive associated values for
the matching queries.</li>
</ul></div>
<div class="section">
<h3><a name="Example_Configuration"></a>Example Configuration</h3>
<p>An example configuration for the YAF sensor is as follows:</p>
-<div class="source">
-<div class="source">
-<pre>{
+<div>
+<div>
+<pre class="source">{
"enrichment": {
"fieldMap": {
"geo": [
@@ -646,6 +449,7 @@
}
}
</pre></div></div>
+
<p>ThreatIntel alert levels are emitted as a new field
“threat.triage.level.” So for the example above, an incoming
message that trips the <tt>ip_src_addr</tt> rule will have a new field
threat.triage.level=10.</p>
<p><a name="Example_Enrichment_via_Stellar"></a></p>
<h1>Example Enrichment via Stellar</h1>
@@ -653,113 +457,95 @@
<div class="section">
<h2><a name="Install_Prerequisites"></a>Install Prerequisites</h2>
<p>Now let’s install some prerequisites:</p>
-
<ul>
-
+
<li>Squid client via <tt>yum install squid</tt></li>
-
<li>ES Head plugin via <tt>/usr/share/elasticsearch/bin/plugin install
mobz/elasticsearch-head</tt></li>
</ul>
<p>Start Squid via <tt>service squid start</tt></p></div>
<div class="section">
<h2><a
name="Adjust_Enrichment_Configurations_for_Squid_to_Call_Stellar"></a>Adjust
Enrichment Configurations for Squid to Call Stellar</h2>
<p>Let’s adjust the configurations for the Squid topology to annotate
the messages using some Stellar functions.</p>
-
<ul>
-
-<li>
-<p>Edit the squid enrichment configuration at
<tt>$METRON_HOME/config/zookeeper/enrichments/squid.json</tt> (this file will
not exist, so create a new one) to add some new fields based on stellar
queries:</p>
-
-<div class="source">
-<div class="source">
-<pre>{
+
+<li>Edit the squid enrichment configuration at
<tt>$METRON_HOME/config/zookeeper/enrichments/squid.json</tt> (this file will
not exist, so create a new one) to add some new fields based on stellar
queries:</li>
+</ul>
+
+<div>
+<div>
+<pre class="source">{
"enrichment" : {
-"fieldMap": {
- "stellar" : {
- "config" : {
- "numeric" : {
- "foo": "1 + 1"
- }
- ,"ALL_CAPS" : "TO_UPPER(source.type)"
- }
- }
- }
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "numeric" : {
+ "foo": "1 + 1"
+ }
+ ,"ALL_CAPS" : "TO_UPPER(source.type)"
+ }
+ }
+ }
},
"threatIntel" : {
-"fieldMap":{
- "stellar" : {
- "config" : {
- "bar" : "TO_UPPER(source.type)"
+ "fieldMap":{
+ "stellar" : {
+ "config" : {
+ "bar" : "TO_UPPER(source.type)"
+ }
+ }
+ },
+ "triageConfig" : {
}
- }
-},
-"triageConfig" : {
-}
}
}
</pre></div></div>
-<p>We have added the following fields as part of the enrichment phase of the
enrichment topology:</p></li>
-
-<li>
-<p><tt>foo</tt> == 2</p></li>
-
+
+<p>We have added the following fields as part of the enrichment phase of the
enrichment topology:</p>
+<ul>
+
+<li><tt>foo</tt> == 2</li>
<li><tt>ALL_CAPS</tt> == SQUID</li>
</ul>
<p>We have added the following as part of the threat intel:</p>
-
<ul>
-
+
<li><tt>bar</tt> == SQUID</li>
</ul>
<p>Please note that foo and ALL_CAPS will be applied in separate workers due
to them being in separate groups.</p>
-
<ul>
-
+
<li>Upload new configs via <tt>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH
-i $METRON_HOME/config/zookeeper -z node1:2181</tt></li>
-
<li>Make the Squid topic in kafka via
<tt>/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181
--create --topic squid --partitions 1 --replication-factor 1</tt></li>
</ul></div>
<div class="section">
<h2><a name="Start_Topologies_and_Send_Data"></a>Start Topologies and Send
Data</h2>
<p>Now we need to start the topologies and send some data:</p>
-
<ul>
-
+
<li>Start the squid topology via <tt>$METRON_HOME/bin/start_parser_topology.sh
-k node1:6667 -z node1:2181 -s squid</tt></li>
-
<li>Generate some data via the squid client:
-
<ul>
-
+
<li><tt>squidclient http://yahoo.com</tt></li>
-
<li><tt>squidclient http://cnn.com</tt></li>
- </ul></li>
-
+</ul>
+</li>
<li>Send the data to kafka via <tt>cat /var/log/squid/access.log |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
node1:6667 --topic squid</tt></li>
-
<li>Browse the data in elasticsearch via the ES Head plugin @ <a
class="externalLink"
href="http://node1:9200/_plugin/head/";>http://node1:9200/_plugin/head/</a> and
verify that in the squid index you have two documents</li>
-
<li>Ensure that the documents have new fields <tt>foo</tt>, <tt>bar</tt> and
<tt>ALL_CAPS</tt> with values as described above.</li>
</ul>
<p>Note that we could have used any Stellar statements here, including calling
out to HBase via <tt>ENRICHMENT_GET</tt> and <tt>ENRICHMENT_EXISTS</tt> or even
calling a machine learning model via <a
href="../../metron-analytics/metron-maas-service/index.html">Model as a
Service</a>.</p></div>
- </div>
- </div>
- </div>
-
+ </div>
+ </div>
+ </div>
<hr/>
-
<footer>
- <div class="container-fluid">
- <div class="row span12">Copyright © 2018
- <a href="https://www.apache.org";>The Apache Software
Foundation</a>.
- All Rights Reserved.
-
+ <div class="container-fluid">
+ <div class="row-fluid">
+é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache,
the Apache feather logo,
+ and the Apache Metron project logo are trademarks of The Apache
Software Foundation.
+ </div>
</div>
-
-
-
- </div>
</footer>
</body>
</html>
