http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-deployment/Kerberos-manual-setup.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/Kerberos-manual-setup.html
b/site/current-book/metron-deployment/Kerberos-manual-setup.html
index 1c1ed0a..229b90a 100644
--- a/site/current-book/metron-deployment/Kerberos-manual-setup.html
+++ b/site/current-book/metron-deployment/Kerberos-manual-setup.html
@@ -1,302 +1,139 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2018-01-03
- | Rendered using Apache Maven Fluido Skin 1.3.0
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from
src/site/markdown/metron-deployment/Kerberos-manual-setup.md at 2018-06-07
+ | Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180103" />
+ <meta name="Date-Revision-yyyymmdd" content="20180607" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Kerberos Setup</title>
- <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.7.min.css" />
<link rel="stylesheet" href="../css/site.css" />
<link rel="stylesheet" href="../css/print.css" media="print" />
-
-
- <script type="text/javascript"
src="../js/apache-maven-fluido-1.3.0.min.js"></script>
-
-
-
-<script type="text/javascript">$( document ).ready( function() { $(
'.carousel' ).carousel( { interval: 3500 } ) } );</script>
-
- </head>
- <body class="topBarDisabled">
-
-
-
-
- <div class="container-fluid">
- <div id="banner">
- <div class="pull-left">
- <a href="http://metron.apache.org/";
id="bannerLeft">
-
<img src="../images/metron-logo.png" alt="Apache Metron"
width="148px" height="48px"/>
- </a>
- </div>
- <div class="pull-right"> </div>
+ <script type="text/javascript"
src="../js/apache-maven-fluido-1.7.min.js"></script>
+<script type="text/javascript">
+ $( document ).ready( function() { $( '.carousel' ).carousel( {
interval: 3500 } ) } );
+ </script>
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href="http://metron.apache.org/";
id="bannerLeft"><img src="../images/metron-logo.png" alt="Apache Metron"
width="148px" height="48px"/></a></div>
+ <div class="pull-right"></div>
<div class="clear"><hr/></div>
</div>
<div id="breadcrumbs">
<ul class="breadcrumb">
-
-
- <li class="">
- <a href="http://www.apache.org"; class="externalLink"
title="Apache">
- Apache</a>
- </li>
- <li class="divider ">/</li>
- <li class="">
- <a href="http://metron.apache.org/"; class="externalLink"
title="Metron">
- Metron</a>
- </li>
- <li class="divider ">/</li>
- <li class="">
- <a href="../index.html" title="Documentation">
- Documentation</a>
- </li>
- <li class="divider ">/</li>
- <li class="">Kerberos Setup</li>
-
-
-
- <li id="publishDate" class="pull-right">Last Published:
2018-01-03</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
-
- </ul>
+ <li class=""><a href="http://www.apache.org"; class="externalLink"
title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="http://metron.apache.org/"; class="externalLink"
title="Metron">Metron</a><span class="divider">/</span></li>
+ <li class=""><a href="../index.html"
title="Documentation">Documentation</a><span class="divider">/</span></li>
+ <li class="active ">Kerberos Setup</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span>
Last Published: 2018-06-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.5.0</li>
+ </ul>
</div>
-
-
<div class="row-fluid">
- <div id="leftColumn" class="span3">
+ <div id="leftColumn" class="span2">
<div class="well sidebar-nav">
-
-
- <ul class="nav nav-list">
- <li class="nav-header">User Documentation</li>
-
- <li>
-
- <a href="../index.html" title="Metron">
- <i class="icon-chevron-down"></i>
- Metron</a>
- <ul class="nav nav-list">
-
- <li>
-
- <a href="../Upgrading.html" title="Upgrading">
- <i class="none"></i>
- Upgrading</a>
- </li>
-
- <li>
-
- <a href="../metron-analytics/index.html"
title="Analytics">
- <i class="icon-chevron-right"></i>
- Analytics</a>
- </li>
-
- <li>
-
- <a href="../metron-contrib/metron-docker/index.html"
title="Docker">
- <i class="none"></i>
- Docker</a>
- </li>
-
- <li>
-
- <a href="../metron-deployment/index.html"
title="Deployment">
- <i class="icon-chevron-down"></i>
- Deployment</a>
- <ul class="nav nav-list">
-
- <li>
-
- <a
href="../metron-deployment/Kerberos-ambari-setup.html"
title="Kerberos-ambari-setup">
- <i class="none"></i>
- Kerberos-ambari-setup</a>
- </li>
-
- <li class="active">
-
- <a href="#"><i class="none"></i>Kerberos-manual-setup</a>
- </li>
-
- <li>
-
- <a href="../metron-deployment/amazon-ec2/index.html"
title="Amazon-ec2">
- <i class="none"></i>
- Amazon-ec2</a>
- </li>
-
- <li>
-
- <a
href="../metron-deployment/other-examples/index.html" title="Other-examples">
- <i class="icon-chevron-right"></i>
- Other-examples</a>
- </li>
-
- <li>
-
- <a
href="../metron-deployment/packaging/ambari/index.html" title="Ambari">
- <i class="none"></i>
- Ambari</a>
- </li>
-
- <li>
-
- <a
href="../metron-deployment/packaging/docker/ansible-docker/index.html"
title="Ansible-docker">
- <i class="none"></i>
- Ansible-docker</a>
- </li>
-
- <li>
-
- <a
href="../metron-deployment/packaging/docker/rpm-docker/index.html"
title="Rpm-docker">
- <i class="none"></i>
- Rpm-docker</a>
- </li>
-
- <li>
-
- <a
href="../metron-deployment/packaging/packer-build/index.html"
title="Packer-build">
- <i class="none"></i>
- Packer-build</a>
- </li>
-
- <li>
-
- <a href="../metron-deployment/roles/index.html"
title="Roles">
- <i class="icon-chevron-right"></i>
- Roles</a>
- </li>
-
- <li>
-
- <a href="../metron-deployment/vagrant/index.html"
title="Vagrant">
- <i class="icon-chevron-right"></i>
- Vagrant</a>
- </li>
- </ul>
- </li>
-
- <li>
-
- <a
href="../metron-interface/metron-alerts/index.html" title="Alerts">
- <i class="none"></i>
- Alerts</a>
- </li>
-
- <li>
-
- <a
href="../metron-interface/metron-config/index.html" title="Config">
- <i class="none"></i>
- Config</a>
- </li>
-
- <li>
-
- <a href="../metron-interface/metron-rest/index.html"
title="Rest">
- <i class="none"></i>
- Rest</a>
- </li>
-
- <li>
-
- <a href="../metron-platform/index.html"
title="Platform">
- <i class="icon-chevron-right"></i>
- Platform</a>
- </li>
-
- <li>
-
- <a href="../metron-sensors/index.html"
title="Sensors">
- <i class="icon-chevron-right"></i>
- Sensors</a>
- </li>
-
- <li>
-
- <a
href="../metron-stellar/stellar-3rd-party-example/index.html"
title="Stellar-3rd-party-example">
- <i class="none"></i>
- Stellar-3rd-party-example</a>
- </li>
-
- <li>
-
- <a
href="../metron-stellar/stellar-common/index.html" title="Stellar-common">
- <i class="icon-chevron-right"></i>
- Stellar-common</a>
- </li>
-
- <li>
-
- <a href="../use-cases/index.html" title="Use-cases">
- <i class="icon-chevron-right"></i>
- Use-cases</a>
- </li>
- </ul>
- </li>
- </ul>
-
-
-
- <hr class="divider" />
-
- <div id="poweredBy">
- <div class="clear"></div>
- <div class="clear"></div>
- <div class="clear"></div>
- <a href="http://maven.apache.org/"; title="Built
by Maven" class="poweredBy">
- <img class="builtBy" alt="Built by Maven"
src="../images/logos/maven-feather.png" />
- </a>
- </div>
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+ <li><a href="../index.html" title="Metron"><span
class="icon-chevron-down"></span>Metron</a>
+ <ul class="nav nav-list">
+ <li><a href="../CONTRIBUTING.html" title="CONTRIBUTING"><span
class="none"></span>CONTRIBUTING</a></li>
+ <li><a href="../Upgrading.html" title="Upgrading"><span
class="none"></span>Upgrading</a></li>
+ <li><a href="../metron-analytics/index.html" title="Analytics"><span
class="icon-chevron-right"></span>Analytics</a></li>
+ <li><a href="../metron-contrib/metron-docker/index.html"
title="Docker"><span class="none"></span>Docker</a></li>
+ <li><a href="../metron-contrib/metron-performance/index.html"
title="Performance"><span class="none"></span>Performance</a></li>
+ <li><a href="../metron-deployment/index.html" title="Deployment"><span
class="icon-chevron-down"></span>Deployment</a>
+ <ul class="nav nav-list">
+ <li><a href="../metron-deployment/Kerberos-ambari-setup.html"
title="Kerberos-ambari-setup"><span
class="none"></span>Kerberos-ambari-setup</a></li>
+ <li class="active"><a href="#"><span
class="none"></span>Kerberos-manual-setup</a></li>
+ <li><a href="../metron-deployment/amazon-ec2/index.html"
title="Amazon-ec2"><span class="none"></span>Amazon-ec2</a></li>
+ <li><a href="../metron-deployment/ansible/index.html"
title="Ansible"><span class="icon-chevron-right"></span>Ansible</a></li>
+ <li><a href="../metron-deployment/development/index.html"
title="Development"><span class="icon-chevron-right"></span>Development</a></li>
+ <li><a href="../metron-deployment/other-examples/index.html"
title="Other-examples"><span
class="icon-chevron-right"></span>Other-examples</a></li>
+ <li><a href="../metron-deployment/packaging/ambari/index.html"
title="Ambari"><span class="icon-chevron-right"></span>Ambari</a></li>
+ <li><a
href="../metron-deployment/packaging/docker/ansible-docker/index.html"
title="Ansible-docker"><span class="none"></span>Ansible-docker</a></li>
+ <li><a href="../metron-deployment/packaging/docker/deb-docker/index.html"
title="Deb-docker"><span class="none"></span>Deb-docker</a></li>
+ <li><a href="../metron-deployment/packaging/docker/rpm-docker/index.html"
title="Rpm-docker"><span class="none"></span>Rpm-docker</a></li>
+ <li><a href="../metron-deployment/packaging/packer-build/index.html"
title="Packer-build"><span class="none"></span>Packer-build</a></li>
+ </ul>
+</li>
+ <li><a href="../metron-interface/metron-alerts/index.html"
title="Alerts"><span class="none"></span>Alerts</a></li>
+ <li><a href="../metron-interface/metron-config/index.html"
title="Config"><span class="none"></span>Config</a></li>
+ <li><a href="../metron-interface/metron-rest/index.html"
title="Rest"><span class="none"></span>Rest</a></li>
+ <li><a href="../metron-platform/index.html" title="Platform"><span
class="icon-chevron-right"></span>Platform</a></li>
+ <li><a href="../metron-sensors/index.html" title="Sensors"><span
class="icon-chevron-right"></span>Sensors</a></li>
+ <li><a href="../metron-stellar/stellar-3rd-party-example/index.html"
title="Stellar-3rd-party-example"><span
class="none"></span>Stellar-3rd-party-example</a></li>
+ <li><a href="../metron-stellar/stellar-common/index.html"
title="Stellar-common"><span
class="icon-chevron-right"></span>Stellar-common</a></li>
+ <li><a href="../metron-stellar/stellar-zeppelin/index.html"
title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
+ <li><a href="../use-cases/index.html" title="Use-cases"><span
class="icon-chevron-right"></span>Use-cases</a></li>
+ </ul>
+</li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href="http://maven.apache.org/"; title="Built by Maven"
class="poweredBy"><img class="builtBy" alt="Built by Maven"
src="../images/logos/maven-feather.png" /></a>
+ </div>
</div>
</div>
-
-
- <div id="bodyColumn" class="span9" >
-
- <h1>Kerberos Setup</h1>
-<p>This document provides instructions for kerberizing Metron’s
Vagrant-based development environments. These instructions do not cover the
Ambari MPack or sensors. General Kerberization notes can be found in the
metron-deployment <a href="../index.html">README.md</a>.</p>
+ <div id="bodyColumn" class="span10" >
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<h1>Kerberos Setup</h1>
+<p>This document provides instructions for kerberizing Metron’s
Vagrant-based development environments. These instructions do not cover the
Ambari MPack or sensors. General Kerberization notes can be found in the
metron-deployment <a href="../index.html">README.md</a>.</p>
<ul>
-
+
<li><a href="#Setup">Setup</a></li>
-
<li><a href="#Setup_a_KDC">Setup a KDC</a></li>
-
<li><a href="#Verify_KDC">Verify KDC</a></li>
-
<li><a href="#Enable_Kerberos">Enable Kerberos</a></li>
-
<li><a href="#Kafka_Authorization">Kafka Authorization</a></li>
-
<li><a href="#HBase_Authorization">HBase Authorization</a></li>
-
<li><a href="#Storm_Authorization">Storm Authorization</a></li>
-
<li><a href="#Start_Metron">Start Metron</a></li>
-
<li><a href="#Push_Data">Push Data</a></li>
-
<li><a href="#More_Information">More Information</a></li>
+<li><a href="#x-pack">Elasticseach X-Pack</a></li>
</ul>
<div class="section">
<h2><a name="Setup"></a>Setup</h2>
-
<ol style="list-style-type: decimal">
-
+
<li>
-<p>Deploy the <a href="vagrant/full-dev-platform/index.html">development
environment.</a>.</p></li>
-
+
+<p>Deploy the <a href="development/centos6/index.html">development
environment.</a>.</p>
+</li>
<li>
-<p>Export the following environment variables. These need to be set for the
remainder of the instructions. Replace <tt>node1</tt> with the appropriate
hosts, if you are running Metron anywhere other than Vagrant.</p>
-
-<div class="source">
-<div class="source">
-<pre># execute as root
+
+<p>Export the following environment variables. These need to be set for the
remainder of the instructions. Replace <tt>node1</tt> with the appropriate
hosts, if you are running Metron anywhere other than Vagrant.</p>
+
+<div>
+<div>
+<pre class="source"># execute as root
sudo su -
export KAFKA_HOME="/usr/hdp/current/kafka-broker"
export ZOOKEEPER=node1:2181
@@ -304,224 +141,244 @@ export ELASTICSEARCH=node1:9200
export BROKERLIST=node1:6667
export HDP_HOME="/usr/hdp/current"
export KAFKA_HOME="${HDP_HOME}/kafka-broker"
-export METRON_VERSION="0.4.2"
+export METRON_VERSION="${METRON_VERSION}"
export METRON_HOME="/usr/metron/${METRON_VERSION}"
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
+
<p>Execute the following commands as root.</p>
-
-<div class="source">
-<div class="source">
-<pre>sudo su -
-</pre></div></div></li>
-
-<li>
-<p>Stop all Metron topologies. They will be restarted again once Kerberos has
been enabled.</p>
-
-<div class="source">
-<div class="source">
-<pre>for topology in bro snort enrichment indexing; do
+
+<div>
+<div>
+<pre class="source">sudo su -
+</pre></div></div>
+</li>
+<li>
+
+<p>Stop all Metron topologies. They will be restarted again once Kerberos has
been enabled.</p>
+
+<div>
+<div>
+<pre class="source">for topology in bro snort enrichment indexing; do
storm kill $topology;
done
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
+
<p>Create the <tt>metron</tt> user’s home directory in HDFS.</p>
-
-<div class="source">
-<div class="source">
-<pre>sudo -u hdfs hdfs dfs -mkdir /user/metron
+
+<div>
+<div>
+<pre class="source">sudo -u hdfs hdfs dfs -mkdir /user/metron
sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron
sudo -u hdfs hdfs dfs -chmod 770 /user/metron
-</pre></div></div></li>
+</pre></div></div>
+</li>
</ol></div>
<div class="section">
<h2><a name="Setup_a_KDC"></a>Setup a KDC</h2>
-
<ol style="list-style-type: decimal">
-
+
<li>
+
<p>Install dependencies.</p>
-
-<div class="source">
-<div class="source">
-<pre>yum -y install krb5-server krb5-libs krb5-workstation
-</pre></div></div></li>
-
+
+<div>
+<div>
+<pre class="source">yum -y install krb5-server krb5-libs krb5-workstation
+</pre></div></div>
+</li>
<li>
+
<p>Define the current host as the KDC.</p>
-
-<div class="source">
-<div class="source">
-<pre>KDC=`hostname`
+
+<div>
+<div>
+<pre class="source">KDC=`hostname`
sed -i.orig 's/kerberos.example.com/'"$KDC"'/g' /etc/krb5.conf
cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
-<p>Ensure that the KDC can issue renewable tickets. This may be necessary on a
real cluster, but should not be on <a
href="vagrant/full-dev-platform/index.html">Full Dev</a>.</p>
+
+<p>Ensure that the KDC can issue renewable tickets. This may be necessary on a
real cluster, but should not be on a <a
href="development/centos6/index.html">single VM</a>.</p>
<p>Edit <tt>/var/kerberos/krb5kdc/kdc.conf</tt> and ensure the following is
added to the <tt>realm</tt> section</p>
-
-<div class="source">
-<div class="source">
-<pre>max_renewable_life = 7d
-</pre></div></div></li>
-
-<li>
-<p>Create the KDC principal database. You will be prompted for a password.
This step takes a moment.</p>
-
-<div class="source">
-<div class="source">
-<pre>kdb5_util create -s
-</pre></div></div></li>
-
+
+<div>
+<div>
+<pre class="source">max_renewable_life = 7d
+</pre></div></div>
+</li>
+<li>
+
+<p>Create the KDC principal database. You will be prompted for a password.
This step takes a moment.</p>
+
+<div>
+<div>
+<pre class="source">kdb5_util create -s
+</pre></div></div>
+</li>
<li>
+
<p>Start the KDC and ensure that it starts on boot.</p>
-
-<div class="source">
-<div class="source">
-<pre>/etc/rc.d/init.d/krb5kdc start
+
+<div>
+<div>
+<pre class="source">/etc/rc.d/init.d/krb5kdc start
chkconfig krb5kdc on
-</pre></div></div></li>
-
-<li>
-<p>Start the Kerberos Admin service and ensure that it starts on boot. </p>
-
-<div class="source">
-<div class="source">
-<pre>/etc/rc.d/init.d/kadmin start
+</pre></div></div>
+</li>
+<li>
+
+<p>Start the Kerberos Admin service and ensure that it starts on boot.</p>
+
+<div>
+<div>
+<pre class="source">/etc/rc.d/init.d/kadmin start
chkconfig kadmin on
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
+
<p>Setup the <tt>admin</tt> principal. You will be prompted for a password; do
not forget it.</p>
-
-<div class="source">
-<div class="source">
-<pre>kadmin.local -q "addprinc admin/admin"
-</pre></div></div></li>
-
+
+<div>
+<div>
+<pre class="source">kadmin.local -q "addprinc admin/admin"
+</pre></div></div>
+</li>
<li>
+
<p>Setup the <tt>metron</tt> principal. You will <tt>kinit</tt> as the
<tt>metron</tt> principal when running topologies. You will be prompted for a
password; do not forget it.</p>
-
-<div class="source">
-<div class="source">
-<pre>kadmin.local -q "addprinc metron"
-</pre></div></div></li>
+
+<div>
+<div>
+<pre class="source">kadmin.local -q "addprinc metron"
+</pre></div></div>
+</li>
</ol></div>
<div class="section">
<h2><a name="Verify_KDC"></a>Verify KDC</h2>
-
<ol style="list-style-type: decimal">
-
+
<li>
+
<p>Ticket renewal is disallowed by default in many Linux distributions. If the
KDC cannot issue renewable tickets, an error will be thrown when starting
Metron’s Storm topologies:</p>
-
-<div class="source">
-<div class="source">
-<pre>Exception in thread "main" java.lang.RuntimeException:
+
+<div>
+<div>
+<pre class="source">Exception in thread "main"
java.lang.RuntimeException:
java.lang.RuntimeException: The TGT found is not renewable
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
-<p>Ensure the Metron keytab is renewable. Look for the ‘R’ flag
in the output of the following command.</p>
-
-<div class="source">
-<div class="source">
-<pre>klist -f
+
+<p>Ensure the Metron keytab is renewable. Look for the ‘R’ flag
in the output of the following command.</p>
+
+<div>
+<div>
+<pre class="source">klist -f
</pre></div></div>
-
+
<ul>
-
+
<li>If the ‘R’ flags are present, you may skip to next
section.</li>
-
<li>If the ‘R’ flags are absent, you will need to follow the
below steps:</li>
- </ul></li>
-
+</ul>
+</li>
<li>
+
<p>If the KDC is already setup, then editing <tt>max_life</tt> and
<tt>max_renewable_life</tt> in <tt>/var/kerberos/krb5kdc/kdc.conf</tt>, then
restarting <tt>kadmin</tt> and <tt>krb5kdc</tt> services will not change the
policies for existing users.</p>
<p>You need to set the renew lifetime for existing users and the
<tt>krbtgt</tt> realm. Modify the appropriate principals to allow renewable
tickets using the following commands. Adjust the parameters to match your
desired KDC parameters:</p>
-
-<div class="source">
-<div class="source">
-<pre>kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days
+allow_renewable krbtgt/example....@example.com"
+
+<div>
+<div>
+<pre class="source">kadmin.local -q "modprinc -maxlife 1days
-maxrenewlife 7days +allow_renewable krbtgt/example....@example.com"
kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days
+allow_renewable met...@example.com"
-</pre></div></div></li>
+</pre></div></div>
+</li>
</ol></div>
<div class="section">
<h2><a name="Enable_Kerberos"></a>Enable Kerberos</h2>
-
<ol style="list-style-type: decimal">
-
+
<li>
+
<p>In <a class="externalLink" href="http://node1:8080";>Ambari</a>, setup Storm
to use Kerberos and run worker jobs as the submitting user.</p>
<p>a. Add the following properties to the custom storm-site:</p>
-
-<div class="source">
-<div class="source">
-<pre>topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
+
+<div>
+<div>
+<pre
class="source">topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
supervisor.run.worker.as.user=true
</pre></div></div>
+
<p>b. In the Storm config section in Ambari, choose “Add
Property” under custom storm-site:</p>
<p><img src="../images/ambari-storm-site.png" alt="custom storm-site" /></p>
<p>c. In the dialog window, choose the “bulk property add mode”
toggle button and add the below values:</p>
-<p><img src="../images/ambari-storm-site-properties.png" alt="custom
storm-site properties" /></p></li>
-
+<p><img src="../images/ambari-storm-site-properties.png" alt="custom
storm-site properties" /></p>
+</li>
<li>
+
<p>Kerberize the cluster via Ambari. More detailed documentation can be found
<a class="externalLink"
href="http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html";>here</a>.</p>
<p>a. For this exercise, choose existing MIT KDC (this is what we setup and
installed in the previous steps.)</p>
<p><img src="../images/enable-kerberos.png" alt="enable keberos" /></p>
<p><img src="../images/enable-kerberos-started.png" alt="enable keberos get
started" /></p>
-<p>b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal
will end up as admin/ad...@example.com when testing the KDC. Use the password
you entered during the step for adding the admin principal.</p>
+<p>b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal
will end up as <a class="externalLink"
href="mailto:admin/ad...@example.com";>admin/ad...@example.com</a> when testing
the KDC. Use the password you entered during the step for adding the admin
principal.</p>
<p><img src="../images/enable-kerberos-configure-kerberos.png" alt="enable
keberos configure" /></p>
<p>c. Click through to “Start and Test Services.” Let the
cluster spin up, but don’t worry about starting up Metron via Ambari -
we’re going to run the parsers manually against the rest of the Hadoop
cluster Kerberized. The wizard will fail at starting Metron, but this is OK.
Click “continue.” When you’re finished, the custom
storm-site should look similar to the following:</p>
-<p><img src="../images/custom-storm-site-final.png" alt="enable keberos
configure" /></p></li>
-
+<p><img src="../images/custom-storm-site-final.png" alt="enable keberos
configure" /></p>
+</li>
<li>
+
<p>Create a Metron keytab</p>
-
-<div class="source">
-<div class="source">
-<pre>kadmin.local -q "ktadd -k metron.headless.keytab
met...@example.com"
+
+<div>
+<div>
+<pre class="source">kadmin.local -q "ktadd -k metron.headless.keytab
met...@example.com"
cp metron.headless.keytab /etc/security/keytabs
chown metron:hadoop /etc/security/keytabs/metron.headless.keytab
chmod 440 /etc/security/keytabs/metron.headless.keytab
-</pre></div></div></li>
+</pre></div></div>
+</li>
</ol></div>
<div class="section">
<h2><a name="Kafka_Authorization"></a>Kafka Authorization</h2>
-
<ol style="list-style-type: decimal">
-
+
<li>
+
<p>Acquire a Kerberos ticket using the <tt>metron</tt> principal.</p>
-
-<div class="source">
-<div class="source">
-<pre>kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
-</pre></div></div></li>
-
-<li>
-<p>Create any additional Kafka topics that you will need. We need to create
the topics before adding the required ACLs. The current full dev installation
will deploy bro, snort, enrichments, and indexing only. For example, you may
want to add a topic for ‘yaf’ telemetry.</p>
-
-<div class="source">
-<div class="source">
-<pre>${KAFKA_HOME}/bin/kafka-topics.sh \
+
+<div>
+<div>
+<pre class="source">kinit -kt /etc/security/keytabs/metron.headless.keytab
met...@example.com
+</pre></div></div>
+</li>
+<li>
+
+<p>Create any additional Kafka topics that you will need. We need to create
the topics before adding the required ACLs. The current full dev installation
will deploy bro, snort, enrichments, and indexing only. For example, you may
want to add a topic for ‘yaf’ telemetry.</p>
+
+<div>
+<div>
+<pre class="source">${KAFKA_HOME}/bin/kafka-topics.sh \
--zookeeper ${ZOOKEEPER} \
--create \
--topic yaf \
--partitions 1 \
--replication-factor 1
-</pre></div></div></li>
-
-<li>
-<p>Setup Kafka ACLs for the <tt>bro</tt>, <tt>snort</tt>,
<tt>enrichments</tt>, and <tt>indexing</tt> topics. Run the same command
against any additional topics that you might be using; for example
<tt>yaf</tt>.</p>
-
-<div class="source">
-<div class="source">
-<pre>export KERB_USER=metron
+</pre></div></div>
+</li>
+<li>
+
+<p>Setup Kafka ACLs for the <tt>bro</tt>, <tt>snort</tt>,
<tt>enrichments</tt>, and <tt>indexing</tt> topics. Run the same command
against any additional topics that you might be using; for example
<tt>yaf</tt>.</p>
+
+<div>
+<div>
+<pre class="source">export KERB_USER=metron
for topic in bro snort enrichments indexing; do
${KAFKA_HOME}/bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
@@ -530,15 +387,16 @@ for topic in bro snort enrichments indexing; do
--allow-principal User:${KERB_USER} \
--topic ${topic}
done
-</pre></div></div></li>
-
-<li>
-<p>Setup Kafka ACLs for the consumer groups. This command sets the ACLs for
Bro, Snort, YAF, Enrichments, Indexing, and the Profiler. Execute the same
command for any additional Parsers that you may be running.</p>
-
-<div class="source">
-<div class="source">
-<pre>export KERB_USER=metron
-for group in bro_parser snort_parser yaf_parser enrichments indexing profiler;
do
+</pre></div></div>
+</li>
+<li>
+
+<p>Setup Kafka ACLs for the consumer groups. This command sets the ACLs for
Bro, Snort, YAF, Enrichments, Indexing, and the Profiler. Execute the same
command for any additional Parsers that you may be running.</p>
+
+<div>
+<div>
+<pre class="source">export KERB_USER=metron
+for group in bro_parser snort_parser yaf_parser enrichments indexing-ra
indexing-batch profiler; do
${KAFKA_HOME}/bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=${ZOOKEEPER} \
@@ -546,84 +404,92 @@ for group in bro_parser snort_parser yaf_parser
enrichments indexing profiler; d
--allow-principal User:${KERB_USER} \
--group ${group}
done
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
+
<p>Add the <tt>metron</tt> principal to the <tt>kafka-cluster</tt> ACL.</p>
-
-<div class="source">
-<div class="source">
-<pre>${KAFKA_HOME}/bin/kafka-acls.sh \
+
+<div>
+<div>
+<pre class="source">${KAFKA_HOME}/bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=${ZOOKEEPER} \
--add \
--allow-principal User:${KERB_USER} \
--cluster kafka-cluster
-</pre></div></div></li>
+</pre></div></div>
+</li>
</ol></div>
<div class="section">
<h2><a name="HBase_Authorization"></a>HBase Authorization</h2>
-
<ol style="list-style-type: decimal">
-
+
<li>
+
<p>Acquire a Kerberos ticket using the <tt>hbase</tt> principal</p>
-
-<div class="source">
-<div class="source">
-<pre>kinit -kt /etc/security/keytabs/hbase.headless.keytab
hbase-metron_clus...@example.com
-</pre></div></div></li>
-
+
+<div>
+<div>
+<pre class="source">kinit -kt /etc/security/keytabs/hbase.headless.keytab
hbase-metron_clus...@example.com
+</pre></div></div>
+</li>
<li>
+
<p>Grant permissions for the HBase tables used in Metron.</p>
-
-<div class="source">
-<div class="source">
-<pre>echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+
+<div>
+<div>
+<pre class="source">echo "grant 'metron', 'RW', 'threatintel'" |
hbase shell
echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
+
<p>If you are using the Profiler, do the same for its HBase table.</p>
-
-<div class="source">
-<div class="source">
-<pre>echo "create 'profiler', 'P'" | hbase shell
+
+<div>
+<div>
+<pre class="source">echo "create 'profiler', 'P'" | hbase shell
echo "grant 'metron', 'RW', 'profiler', 'P'" | hbase shell
-</pre></div></div></li>
+</pre></div></div>
+</li>
</ol></div>
<div class="section">
<h2><a name="Storm_Authorization"></a>Storm Authorization</h2>
-
<ol style="list-style-type: decimal">
-
+
<li>
+
<p>Switch to the <tt>metron</tt> user and acquire a Kerberos ticket for the
<tt>metron</tt> principal.</p>
-
-<div class="source">
-<div class="source">
-<pre>su metron
+
+<div>
+<div>
+<pre class="source">su metron
kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
+
<p>Create the directory <tt>/home/metron/.storm</tt> and switch to that
directory.</p>
-
-<div class="source">
-<div class="source">
-<pre>mkdir /home/metron/.storm
+
+<div>
+<div>
+<pre class="source">mkdir /home/metron/.storm
cd /home/metron/.storm
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
-<p>Ensure the Metron keytab is renewable. See <a href="#Verify_KDC">Verify
KDC</a> above.</p></li>
-
+
+<p>Ensure the Metron keytab is renewable. See <a href="#Verify_KDC">Verify
KDC</a> above.</p>
+</li>
<li>
-<p>Create a client JAAS file at <tt>/home/metron/.storm/client_jaas.conf</tt>.
This should look identical to the Storm client JAAS file located at
<tt>/etc/storm/conf/client_jaas.conf</tt> except for the addition of a
<tt>Client</tt> stanza. The <tt>Client</tt> stanza is used for Zookeeper. All
quotes and semicolons are necessary.</p>
-
-<div class="source">
-<div class="source">
-<pre>cat << EOF > client_jaas.conf
+
+<p>Create a client JAAS file at <tt>/home/metron/.storm/client_jaas.conf</tt>.
This should look identical to the Storm client JAAS file located at
<tt>/etc/storm/conf/client_jaas.conf</tt> except for the addition of a
<tt>Client</tt> stanza. The <tt>Client</tt> stanza is used for Zookeeper. All
quotes and semicolons are necessary.</p>
+
+<div>
+<div>
+<pre class="source">cat << EOF > client_jaas.conf
StormClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
@@ -649,127 +515,132 @@ KafkaClient {
principal="met...@example.com";
};
EOF
-</pre></div></div></li>
-
-<li>
-<p>Create a YAML file at <tt>/home/metron/.storm/storm.yaml</tt>. This should
point to the client JAAS file. Set the array of nimbus hosts accordingly.</p>
-
-<div class="source">
-<div class="source">
-<pre>cat << EOF > /home/metron/.storm/storm.yaml
+</pre></div></div>
+</li>
+<li>
+
+<p>Create a YAML file at <tt>/home/metron/.storm/storm.yaml</tt>. This should
point to the client JAAS file. Set the array of nimbus hosts accordingly.</p>
+
+<div>
+<div>
+<pre class="source">cat << EOF > /home/metron/.storm/storm.yaml
nimbus.seeds : ['node1']
java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
storm.thrift.transport :
'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
EOF
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
+
<p>Create an auxiliary storm configuration file at
<tt>/home/metron/storm-config.json</tt>. Note the login config option in the
file points to the client JAAS file.</p>
-
-<div class="source">
-<div class="source">
-<pre>cat << EOF > /home/metron/storm-config.json
+
+<div>
+<div>
+<pre class="source">cat << EOF > /home/metron/storm-config.json
{
"topology.worker.childopts" :
"-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
}
EOF
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
-<p>Configure the Enrichment, Indexing and Profiler topologies to use the
client JAAS file. To do this, the following key-value pairs:</p>
-
+
+<p>Configure the Enrichment, Indexing and Profiler topologies to use the
client JAAS file. To do this, the following key-value pairs:</p>
<ul>
-
+
<li><tt>kafka.security.protocol=PLAINTEXTSASL</tt></li>
-
<li><tt>topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf</tt></li>
- </ul>
+</ul>
<p>must be added to each of the topology properties files:</p>
-
<ul>
-
+
<li><tt>${METRON_HOME}/config/enrichment.properties</tt></li>
-
<li><tt>${METRON_HOME}/config/elasticsearch.properties</tt></li>
-
<li><tt>${METRON_HOME}/config/profiler.properties</tt></li>
- </ul>
+</ul>
<p>You may use the following command to automate this step:</p>
-
-<div class="source">
-<div class="source">
-<pre>for file in enrichment.properties elasticsearch.properties
profiler.properties; do
+
+<div>
+<div>
+<pre class="source">for file in enrichment.properties elasticsearch.properties
profiler.properties; do
echo ${file}
sed -i
"s/^kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/"
"${METRON_HOME}/config/${file}"
sed -i
"s/^topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/"
"${METRON_HOME}/config/${file}"
done
-</pre></div></div></li>
+</pre></div></div>
+</li>
</ol></div>
<div class="section">
<h2><a name="Start_Metron"></a>Start Metron</h2>
-
<ol style="list-style-type: decimal">
-
+
<li>
+
<p>Switch to the <tt>metron</tt> user and acquire a Kerberos ticket for the
<tt>metron</tt> principal.</p>
-
-<div class="source">
-<div class="source">
-<pre>su metron
+
+<div>
+<div>
+<pre class="source">su metron
kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
-</pre></div></div></li>
-
-<li>
-<p>Restart the parser topologies. Be sure to pass in the new parameter,
<tt>-ksp</tt> or <tt>--kafka_security_protocol</tt>. The following command will
start only the Bro and Snort topologies. Execute the same command for any other
Parsers that you may need, for example <tt>yaf</tt>.</p>
-
-<div class="source">
-<div class="source">
-<pre>for parser in bro snort; do
+</pre></div></div>
+</li>
+<li>
+
+<p>Restart the parser topologies. Be sure to pass in the new parameter,
<tt>-ksp</tt> or <tt>--kafka_security_protocol</tt>. The following command
will start only the Bro and Snort topologies. Execute the same command for any
other Parsers that you may need, for example <tt>yaf</tt>.</p>
+
+<div>
+<div>
+<pre class="source">for parser in bro snort; do
${METRON_HOME}/bin/start_parser_topology.sh \
-z ${ZOOKEEPER} \
-s ${parser} \
-ksp SASL_PLAINTEXT \
-e /home/metron/storm-config.json;
done
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
+
<p>Restart the Enrichment and Indexing topologies.</p>
-
-<div class="source">
-<div class="source">
-<pre>${METRON_HOME}/bin/start_enrichment_topology.sh
+
+<div>
+<div>
+<pre class="source">${METRON_HOME}/bin/start_enrichment_topology.sh
${METRON_HOME}/bin/start_elasticsearch_topology.sh
-</pre></div></div></li>
+</pre></div></div>
+</li>
</ol>
<p>Metron should be ready to receive data.</p></div>
<div class="section">
<h2><a name="Push_Data"></a>Push Data</h2>
-
<ol style="list-style-type: decimal">
-
+
<li>
+
<p>Push some sample data to one of the parser topics. E.g for Bro we took raw
data from <a
href="../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput/index.html">metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput</a></p>
-
-<div class="source">
-<div class="source">
-<pre>cat sample-bro.txt |
${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
+
+<div>
+<div>
+<pre class="source">cat sample-bro.txt |
${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
--broker-list ${BROKERLIST} \
--security-protocol SASL_PLAINTEXT \
--topic bro
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
+
<p>Wait a few moments for data to flow through the system and then check for
data in the Elasticsearch indices. Replace yaf with whichever parser type
you’ve chosen.</p>
-
-<div class="source">
-<div class="source">
-<pre>curl -XGET "${ELASTICSEARCH}/bro*/_search"
+
+<div>
+<div>
+<pre class="source">curl -XGET "${ELASTICSEARCH}/bro*/_search"
curl -XGET "${ELASTICSEARCH}/bro*/_count"
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
-<p>You should have data flowing from the parsers all the way through to the
indexes. This completes the Kerberization instructions</p></li>
+
+<p>You should have data flowing from the parsers all the way through to the
indexes. This completes the Kerberization instructions</p>
+</li>
</ol></div>
<div class="section">
<h2><a name="More_Information"></a>More Information</h2>
@@ -777,15 +648,16 @@ curl -XGET "${ELASTICSEARCH}/bro*/_count"
<h3><a name="Kerberos"></a>Kerberos</h3>
<p>Unsure of your Kerberos principal associated with a keytab? There are a
couple ways to get this. One is via the list of principals that Ambari provides
via downloadable csv. If you didn’t download this list, you can also
check the principal manually by running the following against the keytab.</p>
-<div class="source">
-<div class="source">
-<pre>klist -kt /etc/security/keytabs/<keytab-file-name>
+<div>
+<div>
+<pre class="source">klist -kt /etc/security/keytabs/<keytab-file-name>
</pre></div></div>
+
<p>E.g.</p>
-<div class="source">
-<div class="source">
-<pre>klist -kt /etc/security/keytabs/hbase.headless.keytab
+<div>
+<div>
+<pre class="source">klist -kt /etc/security/keytabs/hbase.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hbase.headless.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
@@ -794,16 +666,17 @@ KVNO Timestamp Principal
1 03/28/17 19:29:36 hbase-metron_clus...@example.com
1 03/28/17 19:29:36 hbase-metron_clus...@example.com
1 03/28/17 19:29:36 hbase-metron_clus...@example.com
-</pre></div></div></div>
+</pre></div></div>
+</div>
<div class="section">
<h3><a name="Kafka_with_Kerberos_enabled"></a>Kafka with Kerberos enabled</h3>
<div class="section">
<h4><a name="Running_Sensors"></a>Running Sensors</h4>
<p>A couple steps are required to produce data to a Kerberized Kafka topic. On
the host you’ll be setting up your sensor(s), switch to the metron user
and create a client_jaas.conf file in the metron home directory if one
doesn’t already exist. It should be owned by metron:metron and contain
at least the following stanza that tells the Kafka client how to interact with
Kerberos:</p>
-<div class="source">
-<div class="source">
-<pre>su - metron
+<div>
+<div>
+<pre class="source">su - metron
cat ${METRON_HOME}/client_jaas.conf
...
KafkaClient {
@@ -816,91 +689,296 @@ KafkaClient {
principal="met...@example.com";
};
</pre></div></div>
+
<p>You’ll also need to set KAFKA_OPTS to tell the Kafka client how to
interact with Kerberos.</p>
-<div class="source">
-<div class="source">
-<pre>export
KAFKA_OPTS="-Djava.security.auth.login.config=${METRON_HOME}/client_jaas.conf"
+<div>
+<div>
+<pre class="source">export
KAFKA_OPTS="-Djava.security.auth.login.config=${METRON_HOME}/client_jaas.conf"
</pre></div></div>
+
<p>For sensors that leverage the Kafka console producer to pipe data into
Metron, e.g. Snort and Yaf, you will need to modify the corresponding sensor
shell scripts or config to append the SASL security protocol property.
<tt>--security-protocol SASL_PLAINTEXT</tt>. Be sure to kinit with the metron
user’s keytab before executing the script that starts the sensor.</p>
<p>More notes can be found in <a
href="../metron-sensors/index.html">metron/metron-sensors/README.md</a></p></div>
<div class="section">
<h4><a name="Write_data_to_a_topic_with_SASL"></a>Write data to a topic with
SASL</h4>
-<div class="source">
-<div class="source">
-<pre>cat sample-yaf.txt | ${KAFKA_HOME}/bin/kafka-console-producer.sh \
+<div>
+<div>
+<pre class="source">cat sample-yaf.txt |
${KAFKA_HOME}/bin/kafka-console-producer.sh \
--broker-list ${BROKERLIST} \
--security-protocol PLAINTEXTSASL \
--topic yaf
-</pre></div></div></div>
+</pre></div></div>
+</div>
<div class="section">
<h4><a name="View_topic_data_from_latest_offset_with_SASL"></a>View topic data
from latest offset with SASL</h4>
-<div class="source">
-<div class="source">
-<pre>${KAFKA_HOME}/bin/kafka-console-consumer.sh \
+<div>
+<div>
+<pre class="source">${KAFKA_HOME}/bin/kafka-console-consumer.sh \
--zookeeper ${ZOOKEEPER} \
--security-protocol PLAINTEXTSASL \
--topic yaf
-</pre></div></div></div>
+</pre></div></div>
+</div>
<div class="section">
<h4><a name="Modify_the_sensor-stubs_to_send_logs_via_SASL"></a>Modify the
sensor-stubs to send logs via SASL</h4>
-<div class="source">
-<div class="source">
-<pre>sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL
--topic/' /opt/sensor-stubs/bin/start-*-stub
+<div>
+<div>
+<pre class="source">sed -i 's/node1:6667 --topic/node1:6667
--security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
for sensorstub in bro snort; do
service sensor-stubs stop ${sensorstub};
service sensor-stubs start ${sensorstub};
done
-</pre></div></div></div>
+</pre></div></div>
+</div>
<div class="section">
<h4><a name="Model_as_a_Service_on_Kerberos"></a>Model as a Service on
Kerberos</h4>
-<p>MaaS works with kerberos, you have to remember to kinit with the metron
user. There is one small issue out of the box (particularly on vagrant), you
get an error like so when running <tt>$METRON_HOME/bin/maas_service.sh</tt>:</p>
+<p>MaaS works with kerberos, you have to remember to kinit with the metron
user. There is one small issue out of the box (particularly on vagrant), you
get an error like so when running <tt>$METRON_HOME/bin/maas_service.sh</tt>:</p>
-<div class="source">
-<div class="source">
-<pre>Requested user metron is not whitelisted and has id 501,which is below
the minimum allowed 1000.
+<div>
+<div>
+<pre class="source">Requested user metron is not whitelisted and has id
501,which is below the minimum allowed 1000.
</pre></div></div>
-<p>In order to correct this, you should:</p>
+<p>In order to correct this, you should:</p>
<ul>
-
+
<li>Navigate to the Yarn configuration in Ambari</li>
-
<li>Click on “Advanced”</li>
-
<li>Scroll to “Advanced yarn-env”</li>
-
<li>Adjust the “Minimum user ID for submitting job” config to
500 from 1000</li>
-
<li>You should then restart Yarn to have the change take effect.</li>
</ul></div></div>
<div class="section">
<h3><a name="References"></a>References</h3>
-
<ul>
-
+
<li><a class="externalLink"
href="https://github.com/apache/storm/blob/master/SECURITY.md";>https://github.com/apache/storm/blob/master/SECURITY.md</a></li>
</ul></div></div>
- </div>
- </div>
- </div>
+<div class="section">
+<h2><a name="X-Pack"></a>X-Pack</h2>
+<p>First, stop the random_access_indexing topology through the Storm UI or
from the CLI, e.g.</p>
- <hr/>
+<div>
+<div>
+<pre class="source">storm kill random_access_indexing
+</pre></div></div>
+
+<p>Here are instructions for enabling X-Pack with Elasticsearch and Kibana: <a
class="externalLink"
href="https://www.elastic.co/guide/en/x-pack/5.6/installing-xpack.html";>https://www.elastic.co/guide/en/x-pack/5.6/installing-xpack.html</a></p>
+<p>You need to be sure to add the appropriate username and password for
Elasticsearch and Kibana to enable external connections from Metron components.
e.g. the following will create a user “transport_client_user”
with password “changeme” and “superuser”
credentials.</p>
+
+<div>
+<div>
+<pre class="source">sudo /usr/share/elasticsearch/bin/x-pack/users useradd
transport_client_user -p changeme -r superuser
+</pre></div></div>
+<p>Once you’ve picked a password to connect to ES, you need to upload a
1-line file to HDFS with that password in it. Metron will use this file to
securely read the password in order to connect to ES securely.</p>
+<p>Here is an example using “changeme” as the password</p>
+
+<div>
+<div>
+<pre class="source">echo changeme > /tmp/xpack-password
+sudo -u hdfs hdfs dfs -mkdir /apps/metron/elasticsearch/
+sudo -u hdfs hdfs dfs -put /tmp/xpack-password /apps/metron/elasticsearch/
+sudo -u hdfs hdfs dfs -chown metron:metron
/apps/metron/elasticsearch/xpack-password
+</pre></div></div>
+
+<p>New settings have been added to configure the Elasticsearch client. By
default the client will run as the normal ES prebuilt transport client. If you
enable X-Pack you should set the es.client.class as shown below.</p>
+<p>Add the es settings to global.json</p>
+
+<div>
+<div>
+<pre class="source">/usr/metron/0.5.0/config/zookeeper/global.json ->
+
+ "es.client.settings" : {
+ "es.client.class" :
"org.elasticsearch.xpack.client.PreBuiltXPackTransportClient",
+ "es.xpack.username" : "transport_client_user",
+ "es.xpack.password.file" :
"/apps/metron/elasticsearch/xpack-password"
+ }
+</pre></div></div>
+
+<p>Submit the update to Zookeeper</p>
+
+<div>
+<div>
+<pre class="source">$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i
METRON_HOME/config/zookeeper/ -z $ZOOKEEPER
+</pre></div></div>
+
+<p>The last step before restarting the topology is to create a custom X-Pack
shaded and relocated jar. This is up to you because of licensing restrictions,
but here is a sample Maven pom file that should help.</p>
+
+<div>
+<div>
+<pre class="source"><?xml version="1.0"
encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software
+ Foundation (ASF) under one or more contributor license agreements. See
the
+ NOTICE file distributed with this work for additional information
regarding
+ copyright ownership. The ASF licenses this file to You under the Apache
License,
+ Version 2.0 (the "License"); you may not use this file except
in compliance
+ with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
distributed
+ under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES
+ OR CONDITIONS OF ANY KIND, either express or implied. See the License
for
+ the specific language governing permissions and limitations under the
License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.elasticsearch</groupId>
+ <artifactId>elasticsearch-xpack-shaded</artifactId>
+ <name>elasticsearch-xpack-shaded</name>
+ <packaging>jar</packaging>
+ <version>5.6.2</version>
+ <repositories>
+ <repository>
+ <id>elasticsearch-releases</id>
+ <url>https://artifacts.elastic.co/maven</url>
+ <releases>
+ <enabled>true</enabled>
+ </releases>
+ <snapshots>
+ <enabled>false</enabled>
+ </snapshots>
+ </repository>
+ </repositories>
+ <dependencies>
+ <dependency>
+ <groupId>org.elasticsearch.client</groupId>
+ <artifactId>x-pack-transport</artifactId>
+ <version>5.6.2</version>
+ <exclusions>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.dataformat</groupId>
+ <artifactId>jackson-dataformat-smile</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.dataformat</groupId>
+ <artifactId>jackson-dataformat-yaml</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.dataformat</groupId>
+ <artifactId>jackson-dataformat-cbor</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <exclusion> <!-- this is causing a weird build error if
not excluded - Error creating shaded jar: null: IllegalArgumentException -->
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-api</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>2.4.3</version>
+ <configuration>
+
<createDependencyReducedPom>true</createDependencyReducedPom>
+ </configuration>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ <relocations>
+ <relocation>
+ <pattern>io.netty</pattern>
+
<shadedPattern>org.apache.metron.io.netty</shadedPattern>
+ </relocation>
+ <relocation>
+
<pattern>org.apache.logging.log4j</pattern>
+
<shadedPattern>org.apache.metron.logging.log4j</shadedPattern>
+ </relocation>
+ </relocations>
+ <artifactSet>
+ <excludes>
+
<exclude>org.slf4j.impl*</exclude>
+
<exclude>org.slf4j:slf4j-log4j*</exclude>
+ </excludes>
+ </artifactSet>
+ <transformers>
+ <transformer
+
implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
+ <resources>
+ <resource>.yaml</resource>
+
<resource>LICENSE.txt</resource>
+ <resource>ASL2.0</resource>
+
<resource>NOTICE.txt</resource>
+ </resources>
+ </transformer>
+ <transformer
+
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
+ <transformer
+
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+ <mainClass></mainClass>
+ </transformer>
+ </transformers>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
+</pre></div></div>
+
+<p>Once you’ve built the elasticsearch-xpack-shaded-5.6.2.jar, it needs
to be made available to Storm when you submit the topology. Create a contrib
directory for indexing and put the jar file in this directory.</p>
+
+<div>
+<div>
+<pre
class="source">/usr/metron/0.5.0/indexing_contrib/elasticsearch-xpack-shaded-5.6.2.jar
+</pre></div></div>
+
+<p>Now you can restart the Elasticsearch topology. Note, you should perform
this step manually, as follows.</p>
+
+<div>
+<div>
+<pre class="source">$METRON_HOME/bin/start_elasticsearch_topology.sh
+</pre></div></div>
+
+<p>Once you’ve performed these steps, you shoud be able to start seeing
data in your ES indexes.</p></div>
+ </div>
+ </div>
+ </div>
+ <hr/>
<footer>
- <div class="container-fluid">
- <div class="row span12">Copyright © 2018
- <a href="https://www.apache.org";>The Apache Software
Foundation</a>.
- All Rights Reserved.
-
+ <div class="container-fluid">
+ <div class="row-fluid">
+é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache,
the Apache feather logo,
+ and the Apache Metron project logo are trademarks of The Apache
Software Foundation.
+ </div>
</div>
-
-
-
- </div>
</footer>
</body>
</html>
