Added: release/metron/0.4.1/site-book/metron-deployment/packaging/ambari/index.html ============================================================================== --- release/metron/0.4.1/site-book/metron-deployment/packaging/ambari/index.html (added) +++ release/metron/0.4.1/site-book/metron-deployment/packaging/ambari/index.html Fri Sep 15 23:37:46 2017 @@ -0,0 +1,786 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-09-08 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170908" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Ambari Management Pack Development</title> + <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../../css/site.css" /> + <link rel="stylesheet" href="../../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Ambari Management Pack Development</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-09-08</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-down"></i> + Deployment</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"> + <i class="none"></i> + Kerberos-ambari-setup</a> + </li> + + <li> + + <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"> + <i class="none"></i> + Kerberos-manual-setup</a> + </li> + + <li> + + <a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"> + <i class="none"></i> + Amazon-ec2</a> + </li> + + <li> + + <a href="../../../metron-deployment/other-examples/index.html" title="Other-examples"> + <i class="icon-chevron-right"></i> + Other-examples</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Ambari</a> + </li> + + <li> + + <a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"> + <i class="none"></i> + Ansible-docker</a> + </li> + + <li> + + <a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"> + <i class="none"></i> + Rpm-docker</a> + </li> + + <li> + + <a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"> + <i class="none"></i> + Packer-build</a> + </li> + + <li> + + <a href="../../../metron-deployment/roles/index.html" title="Roles"> + <i class="icon-chevron-right"></i> + Roles</a> + </li> + + <li> + + <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant"> + <i class="icon-chevron-right"></i> + Vagrant</a> + </li> + </ul> + </li> + + <li> + + <a href="../../../metron-interface/metron-alerts/index.html" title="Alerts"> + <i class="none"></i> + Alerts</a> + </li> + + <li> + + <a href="../../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + + <li> + + <a href="../../../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li> + + <a href="../../../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-right"></i> + Use-cases</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Ambari Management Pack Development</h1> +<p><a name="Ambari_Management_Pack_Development"></a></p> +<p>Typically, Ambari Management Pack development will be done in the Vagrant environments. These instructions are specific to Vagrant, but can be adapted for other environemnts (e.g. make sure to be on the correct nodes for server vs agent files)</p> +<p>There is an <tt>mpack.json</tt> file which describes what services the mpack will contains, versions, etc.</p> +<p>Alongside this are two directories, <tt>addon-services</tt> and <tt>common-services</tt>.</p> +<p>The layout of <tt>/common-services/METRON.CURRENT</tt> is</p> + +<ul> + +<li><tt>/configuration</tt> + +<ul> + +<li>This contains a set of <tt>*-env.xml</tt> files, relevent to particular components or the service as a whole. These are where properties are defined.</li> + </ul></li> + +<li><tt>/package</tt> + +<ul> + +<li><tt>/files</tt> + +<ul> + +<li>Contains files that get used as provided, in particular Elasticsearch templates.</li> + </ul></li> + +<li><tt>/scripts</tt> + +<ul> + +<li>A set of Python scripts that interface with Ambari to manage setup and install</li> + +<li><tt>/params</tt> + +<ul> + +<li>A set of Python scripts for managing parameters from the <tt>*-env.xml</tt> files</li> + </ul></li> + +<li><tt>/templates</tt> + +<ul> + +<li>A set of Jinja template files which can be populated with properties</li> + </ul></li> + </ul></li> + </ul></li> + +<li><tt>/quicklinks</tt> + +<ul> + +<li>Contains <tt>quicklinks.json</tt> to define the Ambari quicklinks that should be used in the UI</li> + </ul></li> + +<li><tt>themes</tt> + +<ul> + +<li>Manages the Ambari UI themes blocks for organizing the configuration by relevant area.</li> + </ul></li> + +<li><tt>kerberos.json</tt> + +<ul> + +<li>Defines the keytabs and other Kerberos configuration to be used when Kerberizing a cluster</li> + </ul></li> + +<li><tt>metainfo.xml</tt> + +<ul> + +<li>Defines the METRON service, along with required packages, services, etc.</li> + </ul></li> + +<li><tt>service_advisor.py</tt> + +<ul> + +<li>Handles component layout and validation, along with handling some configurations for other services or that needs configs from other services.</li> + </ul></li> +</ul> +<p>The layout of <tt>/addon-services/METRON.CURRENT</tt> is</p> + +<ul> + +<li><tt>/repos</tt> + +<ul> + +<li>Contains <tt>repoinfo.xml</tt> that defines repositories to install packages from</li> + </ul></li> + +<li><tt>metainfo.xml</tt> + +<ul> + +<li>Limited info version of <tt>/common-services/METRON.CURRENT/metainfo.xml</tt></li> + </ul></li> + +<li><tt>role_command_order.json</tt> + +<ul> + +<li>Defines the order of service startup and other actions relative to each other.</li> + </ul></li> +</ul> +<div class="section"> +<h2><a name="Adding_a_new_property"></a>Adding a new property</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Add the property to the appropriate <tt>*-env.xml</tt> file found in <tt>METRON.CURRENT/configuration</tt>.</p> + +<div class="source"> +<div class="source"> +<pre><property> + <name>new_property</name> + <description>New Property description</description> + <value>Default Value</value> + <display-name>New Property Pretty Name</display-name> +</property> +</pre></div></div> +<p>The appropriate <tt>*-env.xml</tt> file should be selected based on which component depends on the property. This allows Ambari to accurately restart only the affected components when the property is changed. If a property is in <tt>metron-env.xml</tt>, Ambari will prompt you to restart all Metron components.</p></li> + +<li> +<p>Add the property to the <tt>metron_theme.json</tt> file found in <tt>METRON.CURRENT/themes</tt> if the property was added to a component-specific <tt>*-env.xml</tt> file (<tt>metron-parsers-env.xml</tt> for example) and not <tt>metron-env.xml</tt>. This is necessary for the property to be displayed in the correct tab of the Metron Configs section in the Ambari UI. Using other properties as a guide, add the property to <tt>/configuration/placement/configs</tt> for proper placement and also to <tt>/configuration/widgets</tt> for a specific widget type.</p></li> + +<li> +<p>Reference the property in <tt>METRON.CURRENT/package/scriptes/params/params_linux.py</tt>, unless it will be used in Ambari’s status command. It will be stored in a variable. The name doesn’t have to match, but it’s preferred that it does. Make sure to use replace <tt>metron-env</tt> the correct <tt>*-env</tt> file, as noted above.</p> + +<div class="source"> +<div class="source"> +<pre> new_property = config['configurations']['metron-env']['new_property'] +</pre></div></div> +<p>If this property will be used in the status command, instead make this change in <tt>METRON.CURRENT/package/scriptes/params/status_params.py</tt>. Afterwards, in <tt>params_linux.py</tt>, reference the new property:</p></li> +</ol> + +<div class="source"> +<div class="source"> +<pre> new_property = status_params.new_property +</pre></div></div> +<p>This behavior is because Ambari doesn’t send all parameters to the status, so it needs to be explicitly provided.</p> + +<ol style="list-style-type: decimal"> + +<li> +<p>Ambari master services can then import the params:</p> + +<div class="source"> +<div class="source"> +<pre> from params import params + env.set_params(params) +</pre></div></div></li> + +<li> +<p>The <tt>*_commands.py</tt> files receive the params as an input from the master services. Once this is done, they can be accessed via the variable we set above:</p> + +<div class="source"> +<div class="source"> +<pre> self.__params.new_property +</pre></div></div></li> +</ol> +<div class="section"> +<h3><a name="Env_file_property_walkthrough"></a>Env file property walkthrough</h3> +<p>To illustrate how property files are carried through to the scripts for our services, we’ll run through an existing property, <tt>metron_apps_hdfs_dir</tt>.</p> +<div class="section"> +<h4><a name="Defining_the_property"></a>Defining the property</h4> +<p>First the property appears in the appropriate <tt>*-env.xml</tt>, in this case <tt>METRON.CURRENT/configuration/metron-env.xml</tt>.</p> + +<div class="source"> +<div class="source"> +<pre><property> + <name>metron_apps_hdfs_dir</name> + <value>/apps/metron</value> + <description>Metron apps HDFS dir</description> + <display-name>Metron apps HDFS dir</display-name> +</property> +</pre></div></div> +<p>This defines several things</p> + +<ol style="list-style-type: decimal"> + +<li>The name of the property we’ll be referencing it by in our code.</li> + +<li>The default value of the property.</li> + +<li>The description of the property for the Ambari UI.</li> + +<li>The pretty name that will be shown in Ambari for the property.</li> +</ol></div> +<div class="section"> +<h4><a name="Making_the_property_available_to_scripts"></a>Making the property available to scripts</h4> +<p>Second, we set up the property to be available to the code. This happens in <tt>METRON.CURRENT/packages/scripts/params/params_linux.py</tt>. Just add the following line:</p> + +<div class="source"> +<div class="source"> +<pre>metron_apps_hdfs_dir = config['configurations']['metron-env']['metron_apps_hdfs_dir'] +</pre></div></div> +<p>There is one catch to this. If we wanted this to be available to Ambari’s status command, we’d need to put that line in <tt>METRON.CURRENT/packages/scripts/params/status_params.py</tt> and reference it in <tt>params_linux.py</tt> like so:</p> + +<div class="source"> +<div class="source"> +<pre>metron_apps_hdfs_dir = status_params.metron_apps_hdfs_dir +</pre></div></div> +<p>This behavior is because Ambari doesn’t send all parameters to the status, so it needs to be explicitly provided.</p> +<p>In our case, we don’t use this parameter directly (but it could be if we wanted to use it exactly). We actually append to it before use in <tt>params_linux.py</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>from resource_management.libraries.functions import format +# ... +hdfs_grok_patterns_dir = format("{metron_apps_hdfs_dir}/patterns") +</pre></div></div> +<p>The <tt>format</tt> function is a special Ambari function that will let you create a string with properties in curly braces replaced by their values. In this case, <tt>hdfs_grok_patterns_dir</tt> will be <tt>/apps/metron/patterns</tt> and will be what we now follow in this example.</p></div> +<div class="section"> +<h4><a name="Importing_the_parameters_into_scripts"></a>Importing the parameters into scripts</h4> +<p><tt>hdfs_grok_patterns_dir</tt> is used in <tt>METRON.CURRENT/package/scripts/parser_commands.py</tt>, but before we can reference it, we’ll need the params available to <tt>parser_commands.py</tt></p> +<p>To make them available, we take them in as part of the <tt>__init__</tt></p> + +<div class="source"> +<div class="source"> +<pre>def __init__(self, params): + if params is None: + raise ValueError("params argument is required for initialization") + self.__params = params + # Other initialization +</pre></div></div> +<p>This init is called from various Ambari service methods in <tt>METRON.CURRENT/package/scripts/parser_master.py</tt>, e.g.:</p> + +<div class="source"> +<div class="source"> +<pre>def stop(self, env, upgrade_type=None): + from params import params + env.set_params(params) + commands = ParserCommands(params) + commands.stop_parser_topologies() +</pre></div></div> +<p>Once the params are available to <tt>parser_commands.py</tt>, <tt>hdfs_grok_patterns_dir</tt> is by referencing <tt>self.__params.hdfs_grok_patterns_dir)</tt> In our case, this will create and populate <tt>/apps/metron/patterns</tt> on HDFS, owned by the metron user with appropriate permissions. It’ll also log out what it’s doing, which is important for being able to debug.</p> + +<div class="source"> +<div class="source"> +<pre>def init_parsers(self): + Logger.info( + "Copying grok patterns from local directory '{0}' to HDFS '{1}'".format(self.__params.local_grok_patterns_dir, + self.__params.hdfs_grok_patterns_dir)) + + self.__params.HdfsResource(self.__params.hdfs_grok_patterns_dir, + type="directory", + action="create_on_execute", + owner=self.__params.metron_user, + mode=0755, + source=self.__params.local_grok_patterns_dir) + + Logger.info("Done initializing parser configuration") +</pre></div></div></div></div> +<div class="section"> +<h3><a name="Jinja_Templates_and_properties"></a>Jinja Templates and properties</h3> +<p>Jinja templates allow for the ability to have most of a file defined, but allow variables to be filled in from the properties defined in our <tt>*-env.xml</tt> files.</p> +<p>A variable to be replaced will take the form <tt>{{property_name}}</tt></p> +<p>The properties are made available like any other property, and then the template itself is referenced in Python scripts in the <tt>METRON.CURRENT/package/scripts/</tt> directory.</p></div> +<div class="section"> +<h3><a name="Jinja_template_property_walkthrough"></a>Jinja template property walkthrough</h3> +<p>To illustrate the use of a property in a Jinja template, let’s take an example of an existing property and walk through exactly how it’s implemented.</p> +<p>A straightforward example is <tt>metron_log_dir</tt> in <tt>METRON.CURRENT/configuration/metron-env.xml</tt></p> +<div class="section"> +<h4><a name="Defining_the_property"></a>Defining the property</h4> +<p>First, we need the property in the configuration file:</p> + +<div class="source"> +<div class="source"> +<pre><property> + <name>metron_log_dir</name> + <value>/var/log/metron</value> + <description>Log directory for metron</description> + <display-name>Metron log dir</display-name> +</property> +</pre></div></div></div> +<div class="section"> +<h4><a name="Making_the_property_available_to_templates"></a>Making the property available to templates</h4> +<p>This property isn’t used in Ambari’s status check, so it was directly added to <tt>METRON.CURRENT/package/scripts/params/params_linux.py</tt>. All we do is add the variable, and reference Ambari’s config object appropriately, making sure to reference <tt>metron-env</tt> as the file where the property is located.</p> + +<div class="source"> +<div class="source"> +<pre>metron_log_dir = config['configurations']['metron-env']['metron_log_dir'] +</pre></div></div> +<p>The property is referenced in <tt>metron.j2</tt>.</p> + +<div class="source"> +<div class="source"> +<pre>METRON_LOG_DIR="{{metron_log_dir}}" +</pre></div></div></div> +<div class="section"> +<h4><a name="Using_the_template_in_scripts"></a>Using the template in scripts</h4> +<p>For that property to actually be used, it is referenced in <tt>rest_master.py</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>from resource_management.core.resources.system import File +from resource_management.core.source import Template + +def configure(self, env, upgrade_type=None, config_dir=None): + # Do stuff + + env.set_params(params) + File(format("/etc/sysconfig/metron"), + content=Template("metron.j2") + ) + # Do stuff +</pre></div></div> +<p>This will create a file on the Ambari agent machine’s file system, <tt>/etc/sysconfig/metron</tt>, with the content of <tt>metron.j2</tt>, after replacing <tt>{{metron_log_dir}}</tt> with the value of the property (<tt>/var/log/metron</tt>)</p> + +<div class="source"> +<div class="source"> +<pre>... +METRON_LOG_DIR="/var/log/metron" +... +</pre></div></div></div></div> +<div class="section"> +<h3><a name="Defining_presentation_in_the_Ambari_UI"></a>Defining presentation in the Ambari UI</h3> +<p>Where and how a property is displayed can be controlled in the <tt>METRON.CURRENT/themes/metron_theme.json</tt> file. Consider the <tt>enrichment_workers</tt> property that is defined in a component specific <tt>*-env.xml</tt> file, in this case <tt>METRON.CURRENT/configuration/metron-enrichment-env.xml</tt>. The property appears in <tt>METRON.CURRENT/themes/metron_theme.json</tt> in two different sections:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "configuration": { + "layouts": [...], + "placement": { + "configs": [ + { + "config": "metron-enrichment-env/enrichment_workers", + "subsection-name": "subsection-enrichment-storm" + }, + ... + ] + }, + "widgets": [ + { + "config": "metron-enrichment-env/enrichment_workers", + "widget": { + "type": "text-field" + } + } + ] + } +} +</pre></div></div> +<p>The first setting places the property in the “Storm” section of the “Enrichment” tab in Ambari. Sections are defined in <tt>metron_theme.json</tt> under <tt>/configuration/layouts</tt>.</p> +<p>The second setting defines a widget type of <tt>text-field</tt>. See the <a class="externalLink" href="https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Configs";>Ambari Wiki</a> for more detail on widget types.</p> +<p>If a property is defined in <tt>metron-env.xml</tt>, it is not necessary to add it to the <tt>metron_theme.json</tt> file. By default the property will be located under the “Advanced” tab in the “Advanced metron-env” section.</p></div></div> +<div class="section"> +<h2><a name="How_to_identify_errors_in_MPack_changes"></a>How to identify errors in MPack changes</h2> +<p>Typically, any errors are thrown at one of two times:</p> + +<ol style="list-style-type: decimal"> + +<li>Attempting to install Metron as a whole. These are typically service level definition errors, not property errors. Logs will often be found in <tt>/var/log/ambari-server/ambari-server.log</tt>. Often the log will indicate it was unable to load a particular service or subservice with some more detail.</li> + +<li>Running the actual functionality. These typically tend to cause errors in the UI at runtime. Logs are usually found in <tt>/var/log/ambari-agent/ambari-agent.log</tt>.</li> +</ol> +<p>Unfortunately, because errors tend to occur at runtime, it’s often necessary to add things like logging statements, or even just throw errors to print out in the Ambari UI.</p> +<p>The primary solution to these is to look in the logs for exceptions, see what’s going wrong (Property doesn’t exist? Malformed file couldn’t be loaded?), and adjust appropriately.</p></div> +<div class="section"> +<h2><a name="Testing_changes_without_cycling_Vagrant_build"></a>Testing changes without cycling Vagrant build</h2> +<p>There are techniques we can use to avoid spinning down and spinning back up Vagrant.</p> +<div class="section"> +<h3><a name="Directly_modifying_files_in_Ambari"></a>Directly modifying files in Ambari</h3> +<p>This assumes the installation went through, and we’re just working on getting our particular feature / adjustment to work properly. This is specific to a single node environment like Vagrant, because of other factors (such as consistency across agents). Multinode environments should generally be used when the feature is already stable and updating is preferably a reinstall of the management pack and the Metron service in general.</p> +<p>Ambari stores the Python files from the service in a couple places. We’ll want to update the files, then have Ambari pick up the updated versions and use them as the new basis. A reinstall of Metron is unnecessary for this type of testing.</p> +<p>Specifically, the server files live in</p> + +<div class="source"> +<div class="source"> +<pre>/var/lib/ambari-server/resources/mpacks/metron-ambari.mpack-0.4.0.0/common-services +/var/lib/ambari-agent/cache/common-services +</pre></div></div> +<p>e.g. enrichment_commands.py can be found in:</p> + +<div class="source"> +<div class="source"> +<pre>/var/lib/ambari-server/resources/mpacks/metron-ambari.mpack-0.4.0.0/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py +/var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py +</pre></div></div> +<p>A <tt>find</tt> command can also be useful in quickly locating the exact location of a file, e.g.</p> + +<div class="source"> +<div class="source"> +<pre>[root@node1 ~]# find /var/lib/ -name enrichment_commands.py +/var/lib/ambari-server/resources/mpacks/metron-ambari.mpack-0.4.0.0/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py +/var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py +</pre></div></div> +<p>The steps to update, for anything affecting an Ambari agent node, e.g. setup scripts during the service install process, are:</p> + +<ol style="list-style-type: decimal"> + +<li>Stop Metron through Ambari. If the property in question is used by the topologies, we’ll want them stopped, so they can be restarted with the new property. This can sometimes be skipped if the change doesn’t affect the topologies themselves, but is a case by case choice.</li> + +<li>Edit the file(s) with your changes. The ambari-agent file must be edited, but generally better to update both for consistency.</li> + +<li> +<p>Restart the Ambari Agent to get the cache to pick up the modified file</p> + +<div class="source"> +<div class="source"> +<pre> ambari-agent restart +</pre></div></div></li> + +<li>Start Metron through Ambari if it was stopped.</li> +</ol></div> +<div class="section"> +<h3><a name="Reinstalling_the_mpack"></a>Reinstalling the mpack</h3> +<p>After we’ve modified files in Ambari and the mpack is working, it is a good idea to reinstall it. Fortunately this can be done without rebuilding the Vagrant environment by following these steps:</p> + +<ol style="list-style-type: decimal"> + +<li>Stop Metron through Ambari and remove the Metron service</li> + +<li> +<p>Rebuild the mpack on your local machine and deploy it to Vagrant, ensuring that all changes made directly to files in Ambari were also made in your local environment</p> + +<div class="source"> +<div class="source"> +<pre> cd metron-deployment + mvn clean package + scp packaging/ambari/metron-mpack/target/metron_mpack-0.4.0.0.tar.gz root@node1:~ +</pre></div></div></li> + +<li> +<p>Log in to Vagrant, deploy the mpack and restart Ambari</p> + +<div class="source"> +<div class="source"> +<pre> ssh root@node1 + ambari-server install-mpack --mpack=metron_mpack-0.4.0.0.tar.gz --verbose --force + ambari-server restart +</pre></div></div></li> + +<li>Install the mpack through Ambari as you normally would</li> +</ol></div></div> +<div class="section"> +<h2><a name="Configuration_involving_dependency_services"></a>Configuration involving dependency services</h2> +<p>Metron can define expectations on other services, e.g. Storm’s <tt>topology.classpath</tt> should be <tt>/etc/hbase/conf:/etc/hadoop/conf</tt>. This happens in <tt>METRON.CURRENT/service_advisor.py</tt>.</p> +<p>The value is defined in a map in <tt>getSTORMSiteDesiredValues(self, is_secured)</tt>. This map is used by <tt>validateSTORMSiteConfigurations</tt> to ensure that the value used by the component is what we need it to be. Generally, only the map should need to be modified.</p> +<p>Properties from the other services can also be used and examined. We use this to build the appropriate URL for Storm, because it differs based on Kerberos security. For example, to retrieve from another service, access the configurations array, retrieve the appropriate config file, retrieve the properties, and finally the desired property.</p> + +<div class="source"> +<div class="source"> +<pre>stormUIServerPort = services["configurations"]["storm-site"]["properties"]["ui.port"] +</pre></div></div> +<p>The security of the cluster can be checked with</p> + +<div class="source"> +<div class="source"> +<pre>is_secured = self.isSecurityEnabled(services) +</pre></div></div> +<p>Note that the configuration expectations will be properly enforced if the cluster is upgraded (e.g. the Storm URL is properly updated) and it does not need to occur manually.</p></div> +<div class="section"> +<h2><a name="Kerberos"></a>Kerberos</h2> +<p>Any scripts interacting with underlying secured tech needs to have <tt>kinit</tt> run as needed. This includes anything interacting with HDFS, HBase, Kafka, Storm, etc. This <tt>kinit</tt> should be run in a conditional statement, to avoid breaking non-Kerberized clusters.</p> +<p>Ambari can run <tt>kinit</tt>, as a given user with the appropriate keytab like so:</p> + +<div class="source"> +<div class="source"> +<pre>if self.__params.security_enabled: + metron_security.kinit(self.__params.kinit_path_local, + self.__params.metron_keytab_path, + self.__params.metron_principal_name, + execute_user=self.__params.metron_user) +</pre></div></div> +<p>The <tt>security_enabled</tt> param is already made available, along with appropriate keytabs for metron, hdfs, kafka, and hbase users.</p></div> +<div class="section"> +<h2><a name="Best_practices"></a>Best practices</h2> + +<ul> + +<li> +<p>Write scripts to be idempotent. The pattern currently used is to write a file out when a task is finished, e.g. setting up ACLs or tables. For example, when indexing is configured, a file is written out and checked based on a property.</p> + +<div class="source"> +<div class="source"> +<pre> def set_configured(self): + File(self.__params.indexing_configured_flag_file, + content="", + owner=self.__params.metron_user, + mode=0755) +</pre></div></div> +<p>This is checked in the indexing master</p></li> +</ul> + +<div class="source"> +<div class="source"> +<pre> if not commands.is_configured(): + commands.init_kafka_topics() + commands.init_hdfs_dir() + commands.set_configured() +</pre></div></div> + +<ul> + +<li>Ensure ACLs are properly managed. This includes Kafka and HBase. Often this involves a config file written out as above because this isn’t idempotent! + +<ul> + +<li>Make sure to <tt>kinit</tt> as the correct user for setting up ACLs in a secured cluster. This is usually kafka for Kafka and hbase for HBase.</li> + +<li>See <tt>set_hbase_acls</tt> in <tt>METRON.CURRENT/package/scripts/enrichment_commands.py</tt> for an HBase example</li> + +<li>See <tt>init_kafka_acls</tt> in <tt>METRON.CURRENT/package/scripts/enrichment_commands.py</tt> and <tt>METRON.CURRENT/package/scripts/metron_service.py</tt> for an Kafka example</li> + </ul></li> +</ul></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
Added: release/metron/0.4.1/site-book/metron-deployment/packaging/docker/ansible-docker/index.html ============================================================================== --- release/metron/0.4.1/site-book/metron-deployment/packaging/docker/ansible-docker/index.html (added) +++ release/metron/0.4.1/site-book/metron-deployment/packaging/docker/ansible-docker/index.html Fri Sep 15 23:37:46 2017 @@ -0,0 +1,300 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-09-08 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170908" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Overview</title> + <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../../../css/site.css" /> + <link rel="stylesheet" href="../../../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Overview</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-09-08</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../../../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-down"></i> + Deployment</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"> + <i class="none"></i> + Kerberos-ambari-setup</a> + </li> + + <li> + + <a href="../../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"> + <i class="none"></i> + Kerberos-manual-setup</a> + </li> + + <li> + + <a href="../../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"> + <i class="none"></i> + Amazon-ec2</a> + </li> + + <li> + + <a href="../../../../metron-deployment/other-examples/index.html" title="Other-examples"> + <i class="icon-chevron-right"></i> + Other-examples</a> + </li> + + <li> + + <a href="../../../../metron-deployment/packaging/ambari/index.html" title="Ambari"> + <i class="none"></i> + Ambari</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Ansible-docker</a> + </li> + + <li> + + <a href="../../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"> + <i class="none"></i> + Rpm-docker</a> + </li> + + <li> + + <a href="../../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"> + <i class="none"></i> + Packer-build</a> + </li> + + <li> + + <a href="../../../../metron-deployment/roles/index.html" title="Roles"> + <i class="icon-chevron-right"></i> + Roles</a> + </li> + + <li> + + <a href="../../../../metron-deployment/vagrant/index.html" title="Vagrant"> + <i class="icon-chevron-right"></i> + Vagrant</a> + </li> + </ul> + </li> + + <li> + + <a href="../../../../metron-interface/metron-alerts/index.html" title="Alerts"> + <i class="none"></i> + Alerts</a> + </li> + + <li> + + <a href="../../../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + + <li> + + <a href="../../../../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li> + + <a href="../../../../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-right"></i> + Use-cases</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Overview</h1> +<p><a name="Overview"></a></p> +<p>The Metron ansible-docker container is provided in an effort reduce the installation burden of deploying Metron in a live envirionment. It is provisioned with software required to sucessfully run the deployment scripts.</p> +<div class="section"> +<h2><a name="Building_the_Container"></a>Building the Container</h2> + +<ol style="list-style-type: decimal"> + +<li>Install Docker ( <a class="externalLink" href="https://www.docker.com/products/overview";>https://www.docker.com/products/overview</a> )</li> + +<li>Navigate to <project-directory>/metron-deployment/packaging/docker/ansible-docker</li> + +<li>Build the container <tt>docker build -t ansible-docker:2.0.0.2 .</tt></li> +</ol></div> +<div class="section"> +<h2><a name="Using_the_Container"></a>Using the Container</h2> +<p>Full instructions are found on the wiki at <a class="externalLink" href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361";>https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361</a></p> +<p>tl;dr:</p> + +<ol style="list-style-type: decimal"> + +<li>docker run -it -v <project-directory>:/root/metron ansible-docker:2.0.0.2 bash</li> + +<li>cd /root/metron</li> + +<li>mvn clean package -DskipTests</li> +</ol></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Added: release/metron/0.4.1/site-book/metron-deployment/packaging/docker/rpm-docker/index.html ============================================================================== --- release/metron/0.4.1/site-book/metron-deployment/packaging/docker/rpm-docker/index.html (added) +++ release/metron/0.4.1/site-book/metron-deployment/packaging/docker/rpm-docker/index.html Fri Sep 15 23:37:46 2017 @@ -0,0 +1,287 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-09-08 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170908" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Overview</title> + <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../../../css/site.css" /> + <link rel="stylesheet" href="../../../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Overview</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-09-08</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../../../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-down"></i> + Deployment</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"> + <i class="none"></i> + Kerberos-ambari-setup</a> + </li> + + <li> + + <a href="../../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"> + <i class="none"></i> + Kerberos-manual-setup</a> + </li> + + <li> + + <a href="../../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"> + <i class="none"></i> + Amazon-ec2</a> + </li> + + <li> + + <a href="../../../../metron-deployment/other-examples/index.html" title="Other-examples"> + <i class="icon-chevron-right"></i> + Other-examples</a> + </li> + + <li> + + <a href="../../../../metron-deployment/packaging/ambari/index.html" title="Ambari"> + <i class="none"></i> + Ambari</a> + </li> + + <li> + + <a href="../../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"> + <i class="none"></i> + Ansible-docker</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Rpm-docker</a> + </li> + + <li> + + <a href="../../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"> + <i class="none"></i> + Packer-build</a> + </li> + + <li> + + <a href="../../../../metron-deployment/roles/index.html" title="Roles"> + <i class="icon-chevron-right"></i> + Roles</a> + </li> + + <li> + + <a href="../../../../metron-deployment/vagrant/index.html" title="Vagrant"> + <i class="icon-chevron-right"></i> + Vagrant</a> + </li> + </ul> + </li> + + <li> + + <a href="../../../../metron-interface/metron-alerts/index.html" title="Alerts"> + <i class="none"></i> + Alerts</a> + </li> + + <li> + + <a href="../../../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + + <li> + + <a href="../../../../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li> + + <a href="../../../../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-right"></i> + Use-cases</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Overview</h1> +<p><a name="Overview"></a></p> +<p>The Metron ansible-docker container is provided in an effort reduce the installation burden of deploying Metron in a live envirionment. It is provisioned with software required to sucessfully run the deployment scripts.</p> +<div class="section"> +<h2><a name="Building_the_Container"></a>Building the Container</h2> + +<ol style="list-style-type: decimal"> + +<li>Install Docker ( <a class="externalLink" href="https://www.docker.com/products/overview";>https://www.docker.com/products/overview</a> )</li> + +<li>Navigate to <project-directory>/metron-deployment/packaging/rpm-docker</li> + +<li>Build the container <tt>docker build -t rpm-docker .</tt></li> +</ol></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Added: release/metron/0.4.1/site-book/metron-deployment/packaging/packer-build/index.html ============================================================================== --- release/metron/0.4.1/site-book/metron-deployment/packaging/packer-build/index.html (added) +++ release/metron/0.4.1/site-book/metron-deployment/packaging/packer-build/index.html Fri Sep 15 23:37:46 2017 @@ -0,0 +1,333 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-09-08 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170908" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Build Metron Images</title> + <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../../css/site.css" /> + <link rel="stylesheet" href="../../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Build Metron Images</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-09-08</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-down"></i> + Deployment</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"> + <i class="none"></i> + Kerberos-ambari-setup</a> + </li> + + <li> + + <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"> + <i class="none"></i> + Kerberos-manual-setup</a> + </li> + + <li> + + <a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"> + <i class="none"></i> + Amazon-ec2</a> + </li> + + <li> + + <a href="../../../metron-deployment/other-examples/index.html" title="Other-examples"> + <i class="icon-chevron-right"></i> + Other-examples</a> + </li> + + <li> + + <a href="../../../metron-deployment/packaging/ambari/index.html" title="Ambari"> + <i class="none"></i> + Ambari</a> + </li> + + <li> + + <a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"> + <i class="none"></i> + Ansible-docker</a> + </li> + + <li> + + <a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"> + <i class="none"></i> + Rpm-docker</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Packer-build</a> + </li> + + <li> + + <a href="../../../metron-deployment/roles/index.html" title="Roles"> + <i class="icon-chevron-right"></i> + Roles</a> + </li> + + <li> + + <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant"> + <i class="icon-chevron-right"></i> + Vagrant</a> + </li> + </ul> + </li> + + <li> + + <a href="../../../metron-interface/metron-alerts/index.html" title="Alerts"> + <i class="none"></i> + Alerts</a> + </li> + + <li> + + <a href="../../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + + <li> + + <a href="../../../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li> + + <a href="../../../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-right"></i> + Use-cases</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Build Metron Images</h1> +<p>Based on the fantastic <a class="externalLink" href="https://github.com/chef/bento";>Bento</a> project developed by Chef.</p> +<div class="section"> +<h2><a name="Images_Provided"></a>Images Provided</h2> + +<ul> + +<li>base-centos-6.7: Centos 6.7 + HDP. Used in the full-dev-platform Vagrant image</li> + +<li>quick-dev-centos-6.7: Centos 6.7 + HDP + Metron. Used for the quick-dev-platform Vagrant image.</li> +</ul></div> +<div class="section"> +<h2><a name="Prerequisites"></a>Prerequisites</h2> + +<ul> + +<li><a class="externalLink" href="https://www.packer.io/";>Packer</a> 0.12.2</li> + +<li><a class="externalLink" href="https://www.virtualbox.org/";>Virtualbox</a> 5.0.16+ (Tested with 5.0.20)</li> +</ul> +<p>Build Both Images ———————- Navigate to <your-project-directory>/metron-deployment/packer-build Execute bin/bento build</p> +<p>Packer will build both images and export .box files to the ./builds directory.</p> +<p>Build Single Images ———————- Navigate to <i>your-project-directory</i>/metron-deployment/packer-build</p> + +<ul> + +<li>Base Centos (full-dev)</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>bin/bento build base-centos-6.7.json +</pre></div></div> + +<ul> + +<li>Quick Dev</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>bin/bento build quick-dev-centos-6.7.json +</pre></div></div> +<p>Using Your New Box File ———————- Modify the relevant Vagrantfile (full-dev-platform or quick-dev-platform) replacing the lines:</p> + +<div class="source"> +<div class="source"> +<pre><pre><code>config.vm.box = "<i>box_name</i>" +config.ssh.insert_key = true</code></pre> +</pre></div></div> +<p>with</p> + +<div class="source"> +<div class="source"> +<pre><pre></code>config.vm.box = "<i>test_box_name</i>" +config.vm.box = "<i>PathToBoxfile/Boxfilename</i>" +config.ssh.insert_key = true</code></pre> +</pre></div></div> +<p>Launch the image as usual.</p> +<p>Node: Vagrant will cache boxes, you can force Vagrant to reload your box by running <tt>vagrant box remove <i>test_box_name</i></tt> before launching your new image.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
