http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-sensors/bro-plugin-kafka/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-sensors/bro-plugin-kafka/index.html
b/site/current-book/metron-sensors/bro-plugin-kafka/index.html
deleted file mode 100644
index 638fdcd..0000000
--- a/site/current-book/metron-sensors/bro-plugin-kafka/index.html
+++ /dev/null
@@ -1,519 +0,0 @@
-<!DOCTYPE html>
-<!--
- | Generated by Apache Maven Doxia at 2018-01-03
- | Rendered using Apache Maven Fluido Skin 1.3.0
--->
-<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
- <head>
- <meta charset="UTF-8" />
- <meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180103" />
- <meta http-equiv="Content-Language" content="en" />
- <title>Metron – Logging Bro Output to Kafka</title>
- <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css"
/>
- <link rel="stylesheet" href="../../css/site.css" />
- <link rel="stylesheet" href="../../css/print.css" media="print" />
-
-
- <script type="text/javascript"
src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
-
-
-
-<script type="text/javascript">$( document ).ready( function() { $(
'.carousel' ).carousel( { interval: 3500 } ) } );</script>
-
- </head>
- <body class="topBarDisabled">
-
-
-
-
- <div class="container-fluid">
- <div id="banner">
- <div class="pull-left">
- <a href="http://metron.apache.org/";
id="bannerLeft">
-
<img src="../../images/metron-logo.png" alt="Apache Metron"
width="148px" height="48px"/>
- </a>
- </div>
- <div class="pull-right"> </div>
- <div class="clear"><hr/></div>
- </div>
-
- <div id="breadcrumbs">
- <ul class="breadcrumb">
-
-
- <li class="">
- <a href="http://www.apache.org"; class="externalLink"
title="Apache">
- Apache</a>
- </li>
- <li class="divider ">/</li>
- <li class="">
- <a href="http://metron.apache.org/"; class="externalLink"
title="Metron">
- Metron</a>
- </li>
- <li class="divider ">/</li>
- <li class="">
- <a href="../../index.html" title="Documentation">
- Documentation</a>
- </li>
- <li class="divider ">/</li>
- <li class="">Logging Bro Output to Kafka</li>
-
-
-
- <li id="publishDate" class="pull-right">Last Published:
2018-01-03</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
-
- </ul>
- </div>
-
-
- <div class="row-fluid">
- <div id="leftColumn" class="span3">
- <div class="well sidebar-nav">
-
-
- <ul class="nav nav-list">
- <li class="nav-header">User Documentation</li>
-
- <li>
-
- <a href="../../index.html" title="Metron">
- <i class="icon-chevron-down"></i>
- Metron</a>
- <ul class="nav nav-list">
-
- <li>
-
- <a href="../../Upgrading.html" title="Upgrading">
- <i class="none"></i>
- Upgrading</a>
- </li>
-
- <li>
-
- <a href="../../metron-analytics/index.html"
title="Analytics">
- <i class="icon-chevron-right"></i>
- Analytics</a>
- </li>
-
- <li>
-
- <a
href="../../metron-contrib/metron-docker/index.html" title="Docker">
- <i class="none"></i>
- Docker</a>
- </li>
-
- <li>
-
- <a href="../../metron-deployment/index.html"
title="Deployment">
- <i class="icon-chevron-right"></i>
- Deployment</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-alerts/index.html" title="Alerts">
- <i class="none"></i>
- Alerts</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-config/index.html" title="Config">
- <i class="none"></i>
- Config</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-rest/index.html" title="Rest">
- <i class="none"></i>
- Rest</a>
- </li>
-
- <li>
-
- <a href="../../metron-platform/index.html"
title="Platform">
- <i class="icon-chevron-right"></i>
- Platform</a>
- </li>
-
- <li>
-
- <a href="../../metron-sensors/index.html"
title="Sensors">
- <i class="icon-chevron-down"></i>
- Sensors</a>
- <ul class="nav nav-list">
-
- <li class="active">
-
- <a href="#"><i class="none"></i>Bro-plugin-kafka</a>
- </li>
-
- <li>
-
- <a href="../../metron-sensors/fastcapa/index.html"
title="Fastcapa">
- <i class="none"></i>
- Fastcapa</a>
- </li>
-
- <li>
-
- <a href="../../metron-sensors/pycapa/index.html"
title="Pycapa">
- <i class="none"></i>
- Pycapa</a>
- </li>
- </ul>
- </li>
-
- <li>
-
- <a
href="../../metron-stellar/stellar-3rd-party-example/index.html"
title="Stellar-3rd-party-example">
- <i class="none"></i>
- Stellar-3rd-party-example</a>
- </li>
-
- <li>
-
- <a
href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
- <i class="icon-chevron-right"></i>
- Stellar-common</a>
- </li>
-
- <li>
-
- <a href="../../use-cases/index.html"
title="Use-cases">
- <i class="icon-chevron-right"></i>
- Use-cases</a>
- </li>
- </ul>
- </li>
- </ul>
-
-
-
- <hr class="divider" />
-
- <div id="poweredBy">
- <div class="clear"></div>
- <div class="clear"></div>
- <div class="clear"></div>
- <a href="http://maven.apache.org/"; title="Built
by Maven" class="poweredBy">
- <img class="builtBy" alt="Built by Maven"
src="../../images/logos/maven-feather.png" />
- </a>
- </div>
- </div>
- </div>
-
-
- <div id="bodyColumn" class="span9" >
-
- <h1>Logging Bro Output to Kafka</h1>
-<p>A Bro log writer that sends logging output to Kafka. This provides a
convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and
others, to process the data generated by Bro.</p>
-<div class="section">
-<h2><a name="Installation"></a>Installation</h2>
-
-<ol style="list-style-type: decimal">
-
-<li>
-<p>Install <a class="externalLink"
href="https://github.com/edenhill/librdkafka";>librdkafka</a>, a native client
library for Kafka. This plugin has been tested against the latest release of
librdkafka, which at the time of this writing is v0.9.4.</p>
-<p>In order to use this plugin within a kerberized Kafka environment, you will
also need <tt>libsasl2</tt> installed and will need to pass
<tt>--enable-sasl</tt> to the <tt>configure</tt> script.</p>
-
-<div class="source">
-<div class="source">
-<pre>curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz |
tar xvz
-cd librdkafka-0.9.4/
-./configure --enable-sasl
-make
-sudo make install
-</pre></div></div></li>
-
-<li>
-<p>Build the plugin using the following commands.</p>
-
-<div class="source">
-<div class="source">
-<pre>./configure --bro-dist=$BRO_SRC
-make
-sudo make install
-</pre></div></div></li>
-
-<li>
-<p>Run the following command to ensure that the plugin was installed
successfully.</p>
-
-<div class="source">
-<div class="source">
-<pre>$ bro -N Bro::Kafka
-Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1)
-</pre></div></div></li>
-</ol></div>
-<div class="section">
-<h2><a name="Activation"></a>Activation</h2>
-<p>The following examples highlight different ways that the plugin can be
used. Simply add the Bro script language to your <tt>local.bro</tt> file (for
example, <tt>/usr/share/bro/site/local.bro</tt>) as shown to demonstrate the
example.</p>
-<div class="section">
-<h3><a name="Example_1"></a>Example 1</h3>
-<p>The goal in this example is to send all HTTP and DNS records to a Kafka
topic named <tt>bro</tt>. </p>
-
-<ul>
-
-<li>Any configuration value accepted by librdkafka can be added to the
<tt>kafka_conf</tt> configuration table.</li>
-
-<li>By defining <tt>topic_name</tt> all records will be sent to the same Kafka
topic.</li>
-
-<li>Defining <tt>logs_to_send</tt> will ensure that only HTTP and DNS records
are sent.</li>
-</ul>
-
-<div class="source">
-<div class="source">
-<pre>@load Bro/Kafka/logs-to-kafka.bro
-redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
-redef Kafka::topic_name = "bro";
-redef Kafka::kafka_conf = table(
- ["metadata.broker.list"] = "localhost:9092"
-);
-</pre></div></div></div>
-<div class="section">
-<h3><a name="Example_2"></a>Example 2</h3>
-<p>It is also possible to send each log stream to a uniquely named topic. The
goal in this example is to send all HTTP records to a Kafka topic named
<tt>http</tt> and all DNS records to a separate Kafka topic named
<tt>dns</tt>.</p>
-
-<ul>
-
-<li>The <tt>topic_name</tt> value must be set to an empty string.</li>
-
-<li>The <tt>$path</tt> value of Bro’s Log Writer mechanism is used to
define the topic name.</li>
-
-<li>Any configuration value accepted by librdkafka can be added to the
<tt>$config</tt> configuration table.</li>
-
-<li>Each log writer accepts a separate configuration table.</li>
-</ul>
-
-<div class="source">
-<div class="source">
-<pre>@load Bro/Kafka/logs-to-kafka.bro
-redef Kafka::topic_name = "";
-redef Kafka::tag_json = T;
-
-event bro_init()
-{
- # handles HTTP
- local http_filter: Log::Filter = [
- $name = "kafka-http",
- $writer = Log::WRITER_KAFKAWRITER,
- $config = table(
- ["metadata.broker.list"] = "localhost:9092"
- ),
- $path = "http"
- ];
- Log::add_filter(HTTP::LOG, http_filter);
-
- # handles DNS
- local dns_filter: Log::Filter = [
- $name = "kafka-dns",
- $writer = Log::WRITER_KAFKAWRITER,
- $config = table(
- ["metadata.broker.list"] = "localhost:9092"
- ),
- $path = "dns"
- ];
- Log::add_filter(DNS::LOG, dns_filter);
-}
-</pre></div></div></div>
-<div class="section">
-<h3><a name="Example_3"></a>Example 3</h3>
-<p>You may want to configure bro to filter log messages with certain
characteristics from being sent to your kafka topics. For instance, Metron
currently doesn’t support IPv6 source or destination IPs in the default
enrichments, so it may be helpful to filter those log messages from being sent
to kafka (although there are <a href="#notes">multiple ways</a> to approach
this). In this example we will do that that, and are assuming a somewhat
standard bro kafka plugin configuration, such that:</p>
-
-<ul>
-
-<li>All bro logs are sent to the <tt>bro</tt> topic, by configuring
<tt>Kafka::topic_name</tt>.</li>
-
-<li>Each JSON message is tagged with the appropriate log type (such as
<tt>http</tt>, <tt>dns</tt>, or <tt>conn</tt>), by setting <tt>tag_json</tt> to
true.</li>
-
-<li>If the log message contains a 128 byte long source or destination IP
address, the log is not sent to kafka.</li>
-</ul>
-
-<div class="source">
-<div class="source">
-<pre>@load Bro/Kafka/logs-to-kafka.bro
-redef Kafka::topic_name = "bro";
-redef Kafka::tag_json = T;
-
-event bro_init() &priority=-5
-{
- # handles HTTP
- Log::add_filter(HTTP::LOG, [
- $name = "kafka-http",
- $writer = Log::WRITER_KAFKAWRITER,
- $pred(rec: HTTP::Info) = { return ! (( |rec$id$orig_h| == 128 ||
|rec$id$resp_h| == 128 )); },
- $config = table(
- ["metadata.broker.list"] = "localhost:9092"
- )
- ]);
-
- # handles DNS
- Log::add_filter(DNS::LOG, [
- $name = "kafka-dns",
- $writer = Log::WRITER_KAFKAWRITER,
- $pred(rec: DNS::Info) = { return ! (( |rec$id$orig_h| == 128 ||
|rec$id$resp_h| == 128 )); },
- $config = table(
- ["metadata.broker.list"] = "localhost:9092"
- )
- ]);
-
- # handles Conn
- Log::add_filter(Conn::LOG, [
- $name = "kafka-conn",
- $writer = Log::WRITER_KAFKAWRITER,
- $pred(rec: Conn::Info) = { return ! (( |rec$id$orig_h| == 128 ||
|rec$id$resp_h| == 128 )); },
- $config = table(
- ["metadata.broker.list"] = "localhost:9092"
- )
- ]);
-}
-</pre></div></div>
-<div class="section">
-<h4><a name="Notes"></a>Notes</h4>
-
-<ul>
-
-<li><tt>logs_to_send</tt> is mutually exclusive with <tt>$pred</tt>, thus for
each log you want to set <tt>$pred</tt> on, you must individually setup a
<tt>Log::add_filter</tt> and refrain from including that log in
<tt>logs_to_send</tt>.</li>
-
-<li>You can also filter IPv6 logs from within your Metron cluster <a
href="../../metron-stellar/stellar-common/index.html#IS_IP">using Stellar</a>.
In that case, you wouldn’t apply a predicate in your bro configuration,
and instead Stellar would filter the logs out before they were processed by the
enrichment layer of Metron.</li>
-
-<li>It is also possible to use the <tt>is_v6_subnet()</tt> bro function in
your predicate, as of their <a class="externalLink"
href="https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5";>2.5
release</a>, however the above example should work on <a class="externalLink"
href="https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4";>bro
2.4</a> and newer, which has been the focus of the kafka plugin.</li>
-</ul></div></div></div>
-<div class="section">
-<h2><a name="Settings"></a>Settings</h2>
-<div class="section">
-<h3><a name="kafka_conf"></a><tt>kafka_conf</tt></h3>
-<p>The global configuration settings for Kafka. These values are passed
through directly to librdkafka. Any valid librdkafka settings can be defined in
this table. The full set of valid librdkafka settings are available <a
class="externalLink"
href="https://github.com/edenhill/librdkafka/blob/v0.9.4/CONFIGURATION.md";>here</a>.</p>
-
-<div class="source">
-<div class="source">
-<pre>redef Kafka::kafka_conf = table(
- ["metadata.broker.list"] = "localhost:9092",
- ["client.id"] = "bro"
-);
-</pre></div></div></div>
-<div class="section">
-<h3><a name="topic_name"></a><tt>topic_name</tt></h3>
-<p>The name of the topic in Kafka where all Bro logs will be sent to.</p>
-
-<div class="source">
-<div class="source">
-<pre>redef Kafka::topic_name = "bro";
-</pre></div></div></div>
-<div class="section">
-<h3><a name="max_wait_on_shutdown"></a><tt>max_wait_on_shutdown</tt></h3>
-<p>The maximum number of milliseconds that the plugin will wait for any
backlog of queued messages to be sent to Kafka before forced shutdown.</p>
-
-<div class="source">
-<div class="source">
-<pre>redef Kafka::max_wait_on_shutdown = 3000;
-</pre></div></div></div>
-<div class="section">
-<h3><a name="tag_json"></a><tt>tag_json</tt></h3>
-<p>If true, a log stream identifier is appended to each JSON-formatted
message. For example, a Conn::LOG message will look like <tt>{ 'conn' : { ...
}}</tt>.</p>
-
-<div class="source">
-<div class="source">
-<pre>redef Kafka::tag_json = T;
-</pre></div></div></div>
-<div class="section">
-<h3><a name="debug"></a><tt>debug</tt></h3>
-<p>A comma separated list of debug contexts in librdkafka which you want to
enable. The available contexts are:</p>
-
-<ul>
-
-<li>generic</li>
-
-<li>broker</li>
-
-<li>topic</li>
-
-<li>metadata</li>
-
-<li>queue</li>
-
-<li>msg</li>
-
-<li>protocol</li>
-
-<li>cgrp</li>
-
-<li>security</li>
-
-<li>fetch</li>
-
-<li>feature</li>
-
-<li>all</li>
-</ul></div></div>
-<div class="section">
-<h2><a name="Kerberos"></a>Kerberos</h2>
-<p>This plugin supports producing messages from a kerberized kafka. There are
a couple of prerequisites and a couple of settings to set. </p>
-<div class="section">
-<h3><a name="SASL"></a>SASL</h3>
-<p>If you are using SASL as a security protocol for kafka, then you must have
libsasl or libsasl2 installed. You can tell if sasl is enabled by running the
following from the directory in which you have build librdkafka:</p>
-
-<div class="source">
-<div class="source">
-<pre>examples/rdkafka_example -X builtin.features
-builtin.features = gzip,snappy,ssl,sasl,regex
-</pre></div></div></div>
-<div class="section">
-<h3><a name="Producer_Config"></a>Producer Config</h3>
-<p>As stated above, you can configure the producer kafka configs in
<tt>${BRO_HOME}/share/bro/site/local.bro</tt>. There are a few configs
necessary to set, which are described <a class="externalLink"
href="https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka";>here</a>.
For an environment where the following is true:</p>
-
-<ul>
-
-<li>The broker is <tt>node1:6667</tt></li>
-
-<li>This kafka is using <tt>SASL_PLAINTEXT</tt> as the security protocol</li>
-
-<li>The keytab used is the <tt>metron</tt> keytab</li>
-
-<li>The service principal for <tt>metron</tt> is
<tt>met...@example.com</tt></li>
-</ul>
-<p>The kafka topic <tt>bro</tt> has been given permission for the
<tt>metron</tt> user to write:</p>
-
-<div class="source">
-<div class="source">
-<pre># login using the metron user
-kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
-${KAFKA_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=node1:2181 --add --allow-principal User:metron --topic bro
-</pre></div></div>
-<p>The following is how the <tt>${BRO_HOME}/share/bro/site/local.bro</tt>
looks:</p>
-
-<div class="source">
-<div class="source">
-<pre>@load Bro/Kafka/logs-to-kafka.bro
-redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
-redef Kafka::topic_name = "bro";
-redef Kafka::tag_json = T;
-redef Kafka::kafka_conf = table( ["metadata.broker.list"] =
"node1:6667"
- , ["security.protocol"] =
"SASL_PLAINTEXT"
- , ["sasl.kerberos.keytab"] =
"/etc/security/keytabs/metron.headless.keytab"
- , ["sasl.kerberos.principal"] =
"met...@example.com"
- );
-</pre></div></div></div></div>
- </div>
- </div>
- </div>
-
- <hr/>
-
- <footer>
- <div class="container-fluid">
- <div class="row span12">Copyright © 2018
- <a href="https://www.apache.org";>The Apache Software
Foundation</a>.
- All Rights Reserved.
-
- </div>
-
-
-
- </div>
- </footer>
- </body>
-</html>
