Added: dev/incubator/metron/0.4.0-RC2/book-site/metron-interface/metron-rest/index.html ============================================================================== --- dev/incubator/metron/0.4.0-RC2/book-site/metron-interface/metron-rest/index.html (added) +++ dev/incubator/metron/0.4.0-RC2/book-site/metron-interface/metron-rest/index.html Fri Apr 28 03:53:51 2017 @@ -0,0 +1,1661 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-04-27 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170427" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Metron REST</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Metron REST</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-04-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Rest</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Metron REST</h1> +<p><a name="Metron_REST"></a></p> +<p>This module provides a RESTful API for interacting with Metron.</p> +<div class="section"> +<h2><a name="Prerequisites"></a>Prerequisites</h2> + +<ul> + +<li>A running Metron cluster</li> + +<li>Java 8 installed</li> + +<li>Storm CLI and Metron topology scripts (start_parser_topology.sh, start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed</li> +</ul></div> +<div class="section"> +<h2><a name="Installation"></a>Installation</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Package the application with Maven:</p> + +<div class="source"> +<div class="source"> +<pre> mvn clean package +</pre></div></div></li> + +<li> +<p>Untar the archive in the target directory. The directory structure will look like:</p> + +<div class="source"> +<div class="source"> +<pre> bin +start_metron_rest.sh + lib +metron-rest-$METRON_VERSION.jar +</pre></div></div></li> + +<li> +<p>Create an <tt>application.yml</tt> file with the contents of <a href="src/main/resources/application-vagrant.yml">application-vagrant.yml</a>. Substitute the appropriate Metron service hosts (Kafka, Zookeeper, Storm, etc.) in properties containing <tt>node1</tt> and update the <tt>spring.datasource.*</tt> properties as needed (see the <a href="#Security">Security</a> section for more details).</p></li> + +<li> +<p>Start the application with this command:</p> + +<div class="source"> +<div class="source"> +<pre> ./bin/start_metron_rest.sh /path/to/application.yml +</pre></div></div></li> +</ol></div> +<div class="section"> +<h2><a name="Usage"></a>Usage</h2> +<p>The exposed REST endpoints can be accessed with the Swagger UI at <a class="externalLink" href="http://host:port/swagger-ui.html#/";>http://host:port/swagger-ui.html#/</a>. The default port is 8080 but can be changed in application.yml by setting “server.port” to the desired port.</p></div> +<div class="section"> +<h2><a name="Security"></a>Security</h2> +<p>The metron-rest module uses <a class="externalLink" href="http://projects.spring.io/spring-security/";>Spring Security</a> for authentication and stores user credentials in a relational database. The H2 database is configured by default and is intended only for development purposes. The “dev” profile can be used to automatically load test users:</p> + +<div class="source"> +<div class="source"> +<pre>./bin/start_metron_rest.sh /path/to/application.yml --spring.profiles.active=dev +</pre></div></div> +<p>For <a class="externalLink" href="http://docs.spring.io/spring-boot/docs/1.4.1.RELEASE/reference/htmlsingle/#boot-features-connect-to-production-database";>production use</a>, a relational database should be configured. For example, configuring MySQL would be done as follows:</p> + +<ol style="list-style-type: decimal"> + +<li> +<p>Create a MySQL user for the Metron REST application (<a class="externalLink" href="http://dev.mysql.com/doc/refman/5.7/en/adding-users.html)">http://dev.mysql.com/doc/refman/5.7/en/adding-users.html)</a>.</p></li> + +<li> +<p>Connect to MySQL and create a Metron REST database:</p> + +<div class="source"> +<div class="source"> +<pre> CREATE DATABASE IF NOT EXISTS metronrest +</pre></div></div></li> + +<li> +<p>Add users:</p> + +<div class="source"> +<div class="source"> +<pre> use metronrest; + insert into users (username, password, enabled) values ('your_username','your_password',1); + insert into authorities (username, authority) values ('your_username', 'ROLE_USER'); +</pre></div></div></li> + +<li> +<p>Replace the H2 connection information in the application.yml file with MySQL connection information:</p> + +<div class="source"> +<div class="source"> +<pre> spring: +datasource: + driverClassName: com.mysql.jdbc.Driver + url: jdbc:mysql://mysql_host:3306/metronrest + username: metron_rest_user + password: metron_rest_password + platform: mysql +</pre></div></div></li> + +<li> +<p>Add a dependency for the MySQL JDBC connector in the metron-rest pom.xml:</p> + +<div class="source"> +<div class="source"> +<pre> <dependency> +<groupId>mysql</groupId> +<artifactId>mysql-connector-java</artifactId> +<version>${mysql.client.version}</version> + </dependency> +</pre></div></div></li> + +<li> +<p>Follow the steps in the <a href="#Installation">Installation</a> section</p></li> +</ol></div> +<div class="section"> +<h2><a name="API"></a>API</h2> +<p>Request and Response objects are JSON formatted. The JSON schemas are available in the Swagger UI.</p> + +<table border="0" class="table table-striped"> + <thead> + +<tr class="a"> + +<th> </th> + </tr> + </thead> + <tbody> + +<tr class="b"> + +<td><a href="#GET_apiv1globalconfig"> <tt>GET /api/v1/global/config</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#DELETE_apiv1globalconfig"> <tt>DELETE /api/v1/global/config</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#POST_apiv1globalconfig"> <tt>POST /api/v1/global/config</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1grokgetstatement"> <tt>GET /api/v1/grok/get/statement</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1groklist"> <tt>GET /api/v1/grok/list</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#POST_apiv1grokvalidate"> <tt>POST /api/v1/grok/validate</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#POST_apiv1hdfs"> <tt>POST /api/v1/hdfs</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1hdfs"> <tt>GET /api/v1/hdfs</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#DELETE_apiv1hdfs"> <tt>DELETE /api/v1/hdfs</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1hdfslist"> <tt>GET /api/v1/hdfs/list</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1kafkatopic"> <tt>GET /api/v1/kafka/topic</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#POST_apiv1kafkatopic"> <tt>POST /api/v1/kafka/topic</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1kafkatopicname"> <tt>GET /api/v1/kafka/topic/{name}</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#DELETE_apiv1kafkatopicname"> <tt>DELETE /api/v1/kafka/topic/{name}</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1kafkatopicnamesample"> <tt>GET /api/v1/kafka/topic/{name}/sample</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1sensorenrichmentconfig"> <tt>GET /api/v1/sensor/enrichment/config</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1sensorenrichmentconfiglistavailable"> <tt>GET /api/v1/sensor/enrichment/config/list/available</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#DELETE_apiv1sensorenrichmentconfigname"> <tt>DELETE /api/v1/sensor/enrichment/config/{name}</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#POST_apiv1sensorenrichmentconfigname"> <tt>POST /api/v1/sensor/enrichment/config/{name}</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1sensorenrichmentconfigname"> <tt>GET /api/v1/sensor/enrichment/config/{name}</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1sensorindexingconfig"> <tt>GET /api/v1/sensor/indexing/config</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#DELETE_apiv1sensorindexingconfigname"> <tt>DELETE /api/v1/sensor/indexing/config/{name}</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#POST_apiv1sensorindexingconfigname"> <tt>POST /api/v1/sensor/indexing/config/{name}</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1sensorindexingconfigname"> <tt>GET /api/v1/sensor/indexing/config/{name}</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#POST_apiv1sensorparserconfig"> <tt>POST /api/v1/sensor/parser/config</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1sensorparserconfig"> <tt>GET /api/v1/sensor/parser/config</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1sensorparserconfiglistavailable"> <tt>GET /api/v1/sensor/parser/config/list/available</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#POST_apiv1sensorparserconfigparseMessage"> <tt>POST /api/v1/sensor/parser/config/parseMessage</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1sensorparserconfigreloadavailable"> <tt>GET /api/v1/sensor/parser/config/reload/available</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#DELETE_apiv1sensorparserconfigname"> <tt>DELETE /api/v1/sensor/parser/config/{name}</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1sensorparserconfigname"> <tt>GET /api/v1/sensor/parser/config/{name}</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#POST_apiv1stellarapplytransformations"> <tt>POST /api/v1/stellar/apply/transformations</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1stellarlist"> <tt>GET /api/v1/stellar/list</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1stellarlistfunctions"> <tt>GET /api/v1/stellar/list/functions</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1stellarlistsimplefunctions"> <tt>GET /api/v1/stellar/list/simple/functions</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#POST_apiv1stellarvalidaterules"> <tt>POST /api/v1/stellar/validate/rules</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1storm"> <tt>GET /api/v1/storm</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1stormclientstatus"> <tt>GET /api/v1/storm/client/status</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1stormenrichment"> <tt>GET /api/v1/storm/enrichment</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1stormenrichmentactivate"> <tt>GET /api/v1/storm/enrichment/activate</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1stormenrichmentdeactivate"> <tt>GET /api/v1/storm/enrichment/deactivate</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1stormenrichmentstart"> <tt>GET /api/v1/storm/enrichment/start</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1stormenrichmentstop"> <tt>GET /api/v1/storm/enrichment/stop</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1stormindexing"> <tt>GET /api/v1/storm/indexing</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1stormindexingactivate"> <tt>GET /api/v1/storm/indexing/activate</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1stormindexingdeactivate"> <tt>GET /api/v1/storm/indexing/deactivate</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1stormindexingstart"> <tt>GET /api/v1/storm/indexing/start</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1stormindexingstop"> <tt>GET /api/v1/storm/indexing/stop</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1stormparseractivatename"> <tt>GET /api/v1/storm/parser/activate/{name}</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1stormparserdeactivatename"> <tt>GET /api/v1/storm/parser/deactivate/{name}</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1stormparserstartname"> <tt>GET /api/v1/storm/parser/start/{name}</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1stormparserstopname"> <tt>GET /api/v1/storm/parser/stop/{name}</tt></a></td> + </tr> + +<tr class="b"> + +<td><a href="#GET_apiv1stormname"> <tt>GET /api/v1/storm/{name}</tt></a></td> + </tr> + +<tr class="a"> + +<td><a href="#GET_apiv1user"> <tt>GET /api/v1/user</tt></a></td> + </tr> + </tbody> +</table> +<div class="section"> +<h3><a name="GET_apiv1globalconfig"></a><tt>GET /api/v1/global/config</tt></h3> + +<ul> + +<li>Description: Retrieves the current Global Config from Zookeeper</li> + +<li>Returns: + +<ul> + +<li>200 - Returns current Global Config JSON in Zookeeper</li> + +<li>404 - Global Config JSON was not found in Zookeeper</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="DELETE_apiv1globalconfig"></a><tt>DELETE /api/v1/global/config</tt></h3> + +<ul> + +<li>Description: Deletes the current Global Config from Zookeeper</li> + +<li>Returns: + +<ul> + +<li>200 - Global Config JSON was deleted</li> + +<li>404 - Global Config JSON was not found in Zookeeper</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="POST_apiv1globalconfig"></a><tt>POST /api/v1/global/config</tt></h3> + +<ul> + +<li>Description: Creates or updates the Global Config in Zookeeper</li> + +<li>Input: + +<ul> + +<li>globalConfig - The Global Config JSON to be saved</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Global Config updated. Returns saved Global Config JSON</li> + +<li>201 - Global Config created. Returns saved Global Config JSON</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1grokgetstatement"></a><tt>GET /api/v1/grok/get/statement</tt></h3> + +<ul> + +<li>Description: Retrieves a Grok statement from the classpath</li> + +<li>Input: + +<ul> + +<li>path - Path to classpath resource</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Grok statement</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1groklist"></a><tt>GET /api/v1/grok/list</tt></h3> + +<ul> + +<li>Description: Lists the common Grok statements available in Metron</li> + +<li>Returns: + +<ul> + +<li>200 - JSON object containing pattern label/Grok statements key value pairs</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="POST_apiv1grokvalidate"></a><tt>POST /api/v1/grok/validate</tt></h3> + +<ul> + +<li>Description: Applies a Grok statement to a sample message</li> + +<li>Input: + +<ul> + +<li>grokValidation - Object containing Grok statement and sample message</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - JSON results</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="POST_apiv1hdfs"></a><tt>POST /api/v1/hdfs</tt></h3> + +<ul> + +<li>Description: Writes contents to an HDFS file. Warning: this will overwrite the contents of a file if it already exists.</li> + +<li>Input: + +<ul> + +<li>path - Path to HDFS file</li> + +<li>contents - File contents</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Contents were written</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1hdfs"></a><tt>GET /api/v1/hdfs</tt></h3> + +<ul> + +<li>Description: Reads a file from HDFS and returns the contents</li> + +<li>Input: + +<ul> + +<li>path - Path to HDFS file</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns file contents</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="DELETE_apiv1hdfs"></a><tt>DELETE /api/v1/hdfs</tt></h3> + +<ul> + +<li>Description: Deletes a file from HDFS</li> + +<li>Input: + +<ul> + +<li>path - Path to HDFS file</li> + +<li>recursive - Delete files recursively</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - File was deleted</li> + +<li>404 - File was not found in HDFS</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1hdfslist"></a><tt>GET /api/v1/hdfs/list</tt></h3> + +<ul> + +<li>Description: Reads a file from HDFS and returns the contents</li> + +<li>Input: + +<ul> + +<li>path - Path to HDFS directory</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns file contents</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1kafkatopic"></a><tt>GET /api/v1/kafka/topic</tt></h3> + +<ul> + +<li>Description: Retrieves all Kafka topics</li> + +<li>Returns: + +<ul> + +<li>200 - Returns a list of all Kafka topics</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="POST_apiv1kafkatopic"></a><tt>POST /api/v1/kafka/topic</tt></h3> + +<ul> + +<li>Description: Creates a new Kafka topic</li> + +<li>Input: + +<ul> + +<li>topic - Kafka topic</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns saved Kafka topic</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1kafkatopicname"></a><tt>GET /api/v1/kafka/topic/{name}</tt></h3> + +<ul> + +<li>Description: Retrieves a Kafka topic</li> + +<li>Input: + +<ul> + +<li>name - Kafka topic name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns Kafka topic</li> + +<li>404 - Kafka topic is missing</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="DELETE_apiv1kafkatopicname"></a><tt>DELETE /api/v1/kafka/topic/{name}</tt></h3> + +<ul> + +<li>Description: Deletes a Kafka topic</li> + +<li>Input: + +<ul> + +<li>name - Kafka topic name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Kafka topic was deleted</li> + +<li>404 - Kafka topic is missing</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1kafkatopicnamesample"></a><tt>GET /api/v1/kafka/topic/{name}/sample</tt></h3> + +<ul> + +<li>Description: Retrieves a sample message from a Kafka topic using the most recent offset</li> + +<li>Input: + +<ul> + +<li>name - Kafka topic name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns sample message</li> + +<li>404 - Either Kafka topic is missing or contains no messages</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1sensorenrichmentconfig"></a><tt>GET /api/v1/sensor/enrichment/config</tt></h3> + +<ul> + +<li>Description: Retrieves all SensorEnrichmentConfigs from Zookeeper</li> + +<li>Returns: + +<ul> + +<li>200 - Returns all SensorEnrichmentConfigs</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1sensorenrichmentconfiglistavailable"></a><tt>GET /api/v1/sensor/enrichment/config/list/available</tt></h3> + +<ul> + +<li>Description: Lists the available enrichments</li> + +<li>Returns: + +<ul> + +<li>200 - Returns a list of available enrichments</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="DELETE_apiv1sensorenrichmentconfigname"></a><tt>DELETE /api/v1/sensor/enrichment/config/{name}</tt></h3> + +<ul> + +<li>Description: Deletes a SensorEnrichmentConfig from Zookeeper</li> + +<li>Input: + +<ul> + +<li>name - SensorEnrichmentConfig name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - SensorEnrichmentConfig was deleted</li> + +<li>404 - SensorEnrichmentConfig is missing</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="POST_apiv1sensorenrichmentconfigname"></a><tt>POST /api/v1/sensor/enrichment/config/{name}</tt></h3> + +<ul> + +<li>Description: Updates or creates a SensorEnrichmentConfig in Zookeeper</li> + +<li>Input: + +<ul> + +<li>sensorEnrichmentConfig - SensorEnrichmentConfig</li> + +<li>name - SensorEnrichmentConfig name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - SensorEnrichmentConfig updated. Returns saved SensorEnrichmentConfig</li> + +<li>201 - SensorEnrichmentConfig created. Returns saved SensorEnrichmentConfig</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1sensorenrichmentconfigname"></a><tt>GET /api/v1/sensor/enrichment/config/{name}</tt></h3> + +<ul> + +<li>Description: Retrieves a SensorEnrichmentConfig from Zookeeper</li> + +<li>Input: + +<ul> + +<li>name - SensorEnrichmentConfig name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns SensorEnrichmentConfig</li> + +<li>404 - SensorEnrichmentConfig is missing</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1sensorindexingconfig"></a><tt>GET /api/v1/sensor/indexing/config</tt></h3> + +<ul> + +<li>Description: Retrieves all SensorIndexingConfigs from Zookeeper</li> + +<li>Returns: + +<ul> + +<li>200 - Returns all SensorIndexingConfigs</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="DELETE_apiv1sensorindexingconfigname"></a><tt>DELETE /api/v1/sensor/indexing/config/{name}</tt></h3> + +<ul> + +<li>Description: Deletes a SensorIndexingConfig from Zookeeper</li> + +<li>Input: + +<ul> + +<li>name - SensorIndexingConfig name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - SensorIndexingConfig was deleted</li> + +<li>404 - SensorIndexingConfig is missing</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="POST_apiv1sensorindexingconfigname"></a><tt>POST /api/v1/sensor/indexing/config/{name}</tt></h3> + +<ul> + +<li>Description: Updates or creates a SensorIndexingConfig in Zookeeper</li> + +<li>Input: + +<ul> + +<li>sensorIndexingConfig - SensorIndexingConfig</li> + +<li>name - SensorIndexingConfig name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - SensorIndexingConfig updated. Returns saved SensorIndexingConfig</li> + +<li>201 - SensorIndexingConfig created. Returns saved SensorIndexingConfig</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1sensorindexingconfigname"></a><tt>GET /api/v1/sensor/indexing/config/{name}</tt></h3> + +<ul> + +<li>Description: Retrieves a SensorIndexingConfig from Zookeeper</li> + +<li>Input: + +<ul> + +<li>name - SensorIndexingConfig name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns SensorIndexingConfig</li> + +<li>404 - SensorIndexingConfig is missing</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="POST_apiv1sensorparserconfig"></a><tt>POST /api/v1/sensor/parser/config</tt></h3> + +<ul> + +<li>Description: Updates or creates a SensorParserConfig in Zookeeper</li> + +<li>Input: + +<ul> + +<li>sensorParserConfig - SensorParserConfig</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - SensorParserConfig updated. Returns saved SensorParserConfig</li> + +<li>201 - SensorParserConfig created. Returns saved SensorParserConfig</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1sensorparserconfig"></a><tt>GET /api/v1/sensor/parser/config</tt></h3> + +<ul> + +<li>Description: Retrieves all SensorParserConfigs from Zookeeper</li> + +<li>Returns: + +<ul> + +<li>200 - Returns all SensorParserConfigs</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1sensorparserconfiglistavailable"></a><tt>GET /api/v1/sensor/parser/config/list/available</tt></h3> + +<ul> + +<li>Description: Lists the available parser classes that can be found on the classpath</li> + +<li>Returns: + +<ul> + +<li>200 - Returns a list of available parser classes</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="POST_apiv1sensorparserconfigparseMessage"></a><tt>POST /api/v1/sensor/parser/config/parseMessage</tt></h3> + +<ul> + +<li>Description: Parses a sample message given a SensorParserConfig</li> + +<li>Input: + +<ul> + +<li>parseMessageRequest - Object containing a sample message and SensorParserConfig</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns parsed message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1sensorparserconfigreloadavailable"></a><tt>GET /api/v1/sensor/parser/config/reload/available</tt></h3> + +<ul> + +<li>Description: Scans the classpath for available parser classes and reloads the cached parser class list</li> + +<li>Returns: + +<ul> + +<li>200 - Returns a list of available parser classes</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="DELETE_apiv1sensorparserconfigname"></a><tt>DELETE /api/v1/sensor/parser/config/{name}</tt></h3> + +<ul> + +<li>Description: Deletes a SensorParserConfig from Zookeeper</li> + +<li>Input: + +<ul> + +<li>name - SensorParserConfig name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - SensorParserConfig was deleted</li> + +<li>404 - SensorParserConfig is missing</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1sensorparserconfigname"></a><tt>GET /api/v1/sensor/parser/config/{name}</tt></h3> + +<ul> + +<li>Description: Retrieves a SensorParserConfig from Zookeeper</li> + +<li>Input: + +<ul> + +<li>name - SensorParserConfig name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns SensorParserConfig</li> + +<li>404 - SensorParserConfig is missing</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="POST_apiv1stellarapplytransformations"></a><tt>POST /api/v1/stellar/apply/transformations</tt></h3> + +<ul> + +<li>Description: Executes transformations against a sample message</li> + +<li>Input: + +<ul> + +<li>transformationValidation - Object containing SensorParserConfig and sample message</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns transformation results</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stellarlist"></a><tt>GET /api/v1/stellar/list</tt></h3> + +<ul> + +<li>Description: Retrieves field transformations</li> + +<li>Returns: + +<ul> + +<li>200 - Returns a list field transformations</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stellarlistfunctions"></a><tt>GET /api/v1/stellar/list/functions</tt></h3> + +<ul> + +<li>Description: Lists the Stellar functions that can be found on the classpath</li> + +<li>Returns: + +<ul> + +<li>200 - Returns a list of Stellar functions</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stellarlistsimplefunctions"></a><tt>GET /api/v1/stellar/list/simple/functions</tt></h3> + +<ul> + +<li>Description: Lists the simple Stellar functions (functions with only 1 input) that can be found on the classpath</li> + +<li>Returns: + +<ul> + +<li>200 - Returns a list of simple Stellar functions</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="POST_apiv1stellarvalidaterules"></a><tt>POST /api/v1/stellar/validate/rules</tt></h3> + +<ul> + +<li>Description: Tests Stellar statements to ensure they are well-formed</li> + +<li>Input: + +<ul> + +<li>statements - List of statements to validate</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns validation results</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1storm"></a><tt>GET /api/v1/storm</tt></h3> + +<ul> + +<li>Description: Retrieves the status of all Storm topologies</li> + +<li>Returns: + +<ul> + +<li>200 - Returns a list of topologies with status information</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormclientstatus"></a><tt>GET /api/v1/storm/client/status</tt></h3> + +<ul> + +<li>Description: Retrieves information about the Storm command line client</li> + +<li>Returns: + +<ul> + +<li>200 - Returns storm command line client information</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormenrichment"></a><tt>GET /api/v1/storm/enrichment</tt></h3> + +<ul> + +<li>Description: Retrieves the status of the Storm enrichment topology</li> + +<li>Returns: + +<ul> + +<li>200 - Returns topology status information</li> + +<li>404 - Topology is missing</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormenrichmentactivate"></a><tt>GET /api/v1/storm/enrichment/activate</tt></h3> + +<ul> + +<li>Description: Activates a Storm enrichment topology</li> + +<li>Returns: + +<ul> + +<li>200 - Returns activate response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormenrichmentdeactivate"></a><tt>GET /api/v1/storm/enrichment/deactivate</tt></h3> + +<ul> + +<li>Description: Deactivates a Storm enrichment topology</li> + +<li>Returns: + +<ul> + +<li>200 - Returns deactivate response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormenrichmentstart"></a><tt>GET /api/v1/storm/enrichment/start</tt></h3> + +<ul> + +<li>Description: Starts a Storm enrichment topology</li> + +<li>Returns: + +<ul> + +<li>200 - Returns start response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormenrichmentstop"></a><tt>GET /api/v1/storm/enrichment/stop</tt></h3> + +<ul> + +<li>Description: Stops a Storm enrichment topology</li> + +<li>Input: + +<ul> + +<li>stopNow - Stop the topology immediately</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns stop response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormindexing"></a><tt>GET /api/v1/storm/indexing</tt></h3> + +<ul> + +<li>Description: Retrieves the status of the Storm indexing topology</li> + +<li>Returns: + +<ul> + +<li>200 - Returns topology status information</li> + +<li>404 - Topology is missing</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormindexingactivate"></a><tt>GET /api/v1/storm/indexing/activate</tt></h3> + +<ul> + +<li>Description: Activates a Storm indexing topology</li> + +<li>Returns: + +<ul> + +<li>200 - Returns activate response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormindexingdeactivate"></a><tt>GET /api/v1/storm/indexing/deactivate</tt></h3> + +<ul> + +<li>Description: Deactivates a Storm indexing topology</li> + +<li>Returns: + +<ul> + +<li>200 - Returns deactivate response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormindexingstart"></a><tt>GET /api/v1/storm/indexing/start</tt></h3> + +<ul> + +<li>Description: Starts a Storm indexing topology</li> + +<li>Returns: + +<ul> + +<li>200 - Returns start response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormindexingstop"></a><tt>GET /api/v1/storm/indexing/stop</tt></h3> + +<ul> + +<li>Description: Stops a Storm enrichment topology</li> + +<li>Input: + +<ul> + +<li>stopNow - Stop the topology immediately</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns stop response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormparseractivatename"></a><tt>GET /api/v1/storm/parser/activate/{name}</tt></h3> + +<ul> + +<li>Description: Activates a Storm parser topology</li> + +<li>Input: + +<ul> + +<li>name - Parser name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns activate response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormparserdeactivatename"></a><tt>GET /api/v1/storm/parser/deactivate/{name}</tt></h3> + +<ul> + +<li>Description: Deactivates a Storm parser topology</li> + +<li>Input: + +<ul> + +<li>name - Parser name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns deactivate response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormparserstartname"></a><tt>GET /api/v1/storm/parser/start/{name}</tt></h3> + +<ul> + +<li>Description: Starts a Storm parser topology</li> + +<li>Input: + +<ul> + +<li>name - Parser name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns start response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormparserstopname"></a><tt>GET /api/v1/storm/parser/stop/{name}</tt></h3> + +<ul> + +<li>Description: Stops a Storm parser topology</li> + +<li>Input: + +<ul> + +<li>name - Parser name</li> + +<li>stopNow - Stop the topology immediately</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns stop response message</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1stormname"></a><tt>GET /api/v1/storm/{name}</tt></h3> + +<ul> + +<li>Description: Retrieves the status of a Storm topology</li> + +<li>Input: + +<ul> + +<li>name - Topology name</li> + </ul></li> + +<li>Returns: + +<ul> + +<li>200 - Returns topology status information</li> + +<li>404 - Topology is missing</li> + </ul></li> +</ul></div> +<div class="section"> +<h3><a name="GET_apiv1user"></a><tt>GET /api/v1/user</tt></h3> + +<ul> + +<li>Description: Retrieves the current user</li> + +<li>Returns: + +<ul> + +<li>200 - Current user</li> + </ul></li> +</ul></div></div> +<div class="section"> +<h2><a name="Testing"></a>Testing</h2> +<p>Profiles are includes for both the metron-docker and Quick Dev environments.</p> +<div class="section"> +<h3><a name="metron-docker"></a>metron-docker</h3> +<p>Start the <a href="../../metron-docker/index.html">metron-docker</a> environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:</p> + +<div class="source"> +<div class="source"> +<pre>mvn clean package +mvn spring-boot:run -Drun.profiles=docker,dev +</pre></div></div> +<p>The metron-rest application will be available at <a class="externalLink" href="http://localhost:8080/swagger-ui.html#/";>http://localhost:8080/swagger-ui.html#/</a>.</p></div> +<div class="section"> +<h3><a name="Quick_Dev"></a>Quick Dev</h3> +<p>Start the <a href="../../metron-deployment/vagrant/quick-dev-platform/index.html">Quick Dev</a> environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:</p> + +<div class="source"> +<div class="source"> +<pre>mvn clean package +mvn spring-boot:run -Drun.profiles=vagrant,dev +</pre></div></div> +<p>The metron-rest application will be available at <a class="externalLink" href="http://localhost:8080/swagger-ui.html#/";>http://localhost:8080/swagger-ui.html#/</a>.</p> +<p>To run the application locally on the Quick Dev host, package the application and scp the archive to node1:</p> + +<div class="source"> +<div class="source"> +<pre>mvn clean package +scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:$METRON_HOME +</pre></div></div> +<p>Login to node1 and unarchive the metron-rest application:</p> + +<div class="source"> +<div class="source"> +<pre>ssh root@node1 +cd $METRON_HOME && tar xf ./metron-rest-$METRON_VERSION-archive.tar.gz +</pre></div></div> +<p>Start the application on a different port to avoid conflicting with Ambari:</p> + +<div class="source"> +<div class="source"> +<pre>java -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 +</pre></div></div> +<p>In a cluster with Kerberos enabled, first add metron-rest to the Kafka acls:</p> + +<div class="source"> +<div class="source"> +<pre>sudo su - +export ZOOKEEPER=node1 +export BROKERLIST=node1 +export HDP_HOME="/usr/hdp/current" +export METRON_VERSION="0.4.0" +export METRON_HOME="/usr/metron/${METRON_VERSION}" +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic ambari_kafka_service_check +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic __consumer_offsets +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --group metron-rest +</pre></div></div> +<p>Then start the application as the metron user while including references to the jaas and krb5.confg files and enabling kerberos support:</p> + +<div class="source"> +<div class="source"> +<pre>su metron +cd ~ +java -Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 --kerberos.enabled=true +</pre></div></div> +<p>The metron-rest application will be available at <a class="externalLink" href="http://node1:8082/swagger-ui.html#/";>http://node1:8082/swagger-ui.html#/</a>.</p></div></div> +<div class="section"> +<h2><a name="License"></a>License</h2> +<p>This project depends on the Java Transaction API. See <a class="externalLink" href="https://java.net/projects/jta-spec/";>https://java.net/projects/jta-spec/</a> for more details.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
Added: dev/incubator/metron/0.4.0-RC2/book-site/metron-platform/index.html ============================================================================== --- dev/incubator/metron/0.4.0-RC2/book-site/metron-platform/index.html (added) +++ dev/incubator/metron/0.4.0-RC2/book-site/metron-platform/index.html Fri Apr 28 03:53:51 2017 @@ -0,0 +1,267 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-04-27 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170427" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Current Build</title> + <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../css/site.css" /> + <link rel="stylesheet" href="../css/print.css" media="print" /> + + + <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Current Build</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-04-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li class="active"> + + <a href="#"><i class="icon-chevron-down"></i>Platform</a> + <ul class="nav nav-list"> + + <li> + + <a href="../metron-platform/metron-api/index.html" title="Api"> + <i class="none"></i> + Api</a> + </li> + + <li> + + <a href="../metron-platform/metron-common/index.html" title="Common"> + <i class="icon-chevron-right"></i> + Common</a> + </li> + + <li> + + <a href="../metron-platform/metron-data-management/index.html" title="Data-management"> + <i class="none"></i> + Data-management</a> + </li> + + <li> + + <a href="../metron-platform/metron-enrichment/index.html" title="Enrichment"> + <i class="none"></i> + Enrichment</a> + </li> + + <li> + + <a href="../metron-platform/metron-indexing/index.html" title="Indexing"> + <i class="none"></i> + Indexing</a> + </li> + + <li> + + <a href="../metron-platform/metron-management/index.html" title="Management"> + <i class="none"></i> + Management</a> + </li> + + <li> + + <a href="../metron-platform/metron-parsers/index.html" title="Parsers"> + <i class="none"></i> + Parsers</a> + </li> + + <li> + + <a href="../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"> + <i class="none"></i> + Pcap-backend</a> + </li> + + <li> + + <a href="../metron-platform/metron-writer/index.html" title="Writer"> + <i class="none"></i> + Writer</a> + </li> + </ul> + </li> + + <li> + + <a href="../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <!-- Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to You under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. --><h1>Current Build</h1> +<p><a name="Current_Build"></a></p> +<p>The latest build of metron-platform is 0.4.0.</p> +<p>We are still in the process of merging/porting additional features from our production code base into this open source release. This release will be followed by a number of additional beta releases until the port is complete. We will also work on getting additional documentation and user/developer guides to the community as soon as we can. At this time we offer no support for the beta software, but will try to respond to requests as promptly as we can.</p> +<p><a name="metron-platform"></a></p> +<h1>metron-platform</h1> +<p>Extensible set of Storm topologies and topology attributes for streaming, enriching, indexing, and storing telemetry in Hadoop. General information on Metron is available at <a class="externalLink" href="https://metron.apache.org/";>https://metron.apache.org/</a></p> +<p><a name="Documentation"></a></p> +<h1>Documentation</h1> +<p>Please see documentation within each individual module for description and usage instructions. Sample topologies are provided under Metron_Topologies to get you started with the framework. We pre-assume knowledge of Hadoop, Storm, Kafka, and HBase.</p> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Added: dev/incubator/metron/0.4.0-RC2/book-site/metron-platform/metron-api/index.html ============================================================================== --- dev/incubator/metron/0.4.0-RC2/book-site/metron-platform/metron-api/index.html (added) +++ dev/incubator/metron/0.4.0-RC2/book-site/metron-platform/metron-api/index.html Fri Apr 28 03:53:51 2017 @@ -0,0 +1,303 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-04-27 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170427" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Metron PCAP Service</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Metron PCAP Service</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-04-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-down"></i> + Platform</a> + <ul class="nav nav-list"> + + <li class="active"> + + <a href="#"><i class="none"></i>Api</a> + </li> + + <li> + + <a href="../../metron-platform/metron-common/index.html" title="Common"> + <i class="icon-chevron-right"></i> + Common</a> + </li> + + <li> + + <a href="../../metron-platform/metron-data-management/index.html" title="Data-management"> + <i class="none"></i> + Data-management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"> + <i class="none"></i> + Enrichment</a> + </li> + + <li> + + <a href="../../metron-platform/metron-indexing/index.html" title="Indexing"> + <i class="none"></i> + Indexing</a> + </li> + + <li> + + <a href="../../metron-platform/metron-management/index.html" title="Management"> + <i class="none"></i> + Management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> + <i class="none"></i> + Parsers</a> + </li> + + <li> + + <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"> + <i class="none"></i> + Pcap-backend</a> + </li> + + <li> + + <a href="../../metron-platform/metron-writer/index.html" title="Writer"> + <i class="none"></i> + Writer</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Metron PCAP Service</h1> +<p><a name="Metron_PCAP_Service"></a></p> +<p>The purpose of the Metron PCAP service is to provide a middle tier to negotiate retrieving packet capture data which flows into Metron. This packet data is of a form which <tt>libpcap</tt> based tools can read.</p> +<div class="section"> +<h2><a name="Starting_the_Service"></a>Starting the Service</h2> +<p>You can start the service either via the init.d script installed, <tt>/etc/init.d/pcapservice</tt> or directly via the <tt>yarn jar</tt> command: <tt>yarn jar $METRON_HOME/lib/metron-api-$METRON_VERSION.jar org.apache.metron.pcapservice.rest.PcapService -port $SERVICE_PORT -query_hdfs_path $QUERY_PATH -pcap_hdfs_path $PCAP_PATH</tt></p> +<p>where</p> + +<ul> + +<li><tt>METRON_HOME</tt> is the location of the metron installation</li> + +<li><tt>METRON_VERSION</tt> is the version of the metron installation</li> + +<li><tt>SERVICE_PORT</tt> is the port to bind the REST service to.</li> + +<li><tt>QUERY_PATH</tt> is the temporary location to store query results. They are deleted after the service reads them.</li> + +<li><tt>PCAP_PATH</tt> is the path to the packet data on HDFS</li> +</ul></div> +<div class="section"> +<h2><a name="The_pcapGettergetPcapsByIdentifiers_endpoint"></a>The <tt>/pcapGetter/getPcapsByIdentifiers</tt> endpoint</h2> +<p>This endpoint takes the following query parameters and returns the subset of packets matching this query:</p> + +<ul> + +<li><tt>srcIp</tt> : The source IP to match on</li> + +<li><tt>srcPort</tt> : The source port to match on</li> + +<li><tt>dstIp</tt> : The destination IP to match on</li> + +<li><tt>dstPort</tt> : The destination port to match on</li> + +<li><tt>startTime</tt> : The start time in milliseconds</li> + +<li><tt>endTime</tt> : The end time in milliseconds</li> + +<li><tt>numReducers</tt> : Specify the number of reducers to use when executing the mapreduce job</li> + +<li><tt>includeReverseTraffic</tt> : Indicates if filter should check swapped src/dest addresses and IPs</li> +</ul></div> +<div class="section"> +<h2><a name="The_pcapGettergetPcapsByQuery_endpoint"></a>The <tt>/pcapGetter/getPcapsByQuery</tt> endpoint</h2> +<p>This endpoint takes the following query parameters and returns the subset of packets matching this query. This endpoint exposes Stellar querying capabilities:</p> + +<ul> + +<li><tt>query</tt> : The Stellar query to execute</li> + +<li><tt>startTime</tt> : The start time in milliseconds</li> + +<li><tt>endTime</tt> : The end time in milliseconds</li> + +<li><tt>numReducers</tt> : Specify the number of reducers to use when executing the mapreduce job</li> +</ul> +<p>Example: <tt>curl -XGET "http://node1:8081/pcapGetter/getPcapsByQuery?query=ip_src_addr+==+'192.168.66.121'+and+ip_src_port+==+'60500'&startTime=1476936000000"</tt></p> +<p>All of these parameters are optional. In the case of a missing parameter, it is treated as a wildcard.</p> +<p>Unlike the CLI tool, there is no paging mechanism. The REST API will stream back data as a single file.</p></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> Added: dev/incubator/metron/0.4.0-RC2/book-site/metron-platform/metron-common/3rdPartyStellar.html ============================================================================== --- dev/incubator/metron/0.4.0-RC2/book-site/metron-platform/metron-common/3rdPartyStellar.html (added) +++ dev/incubator/metron/0.4.0-RC2/book-site/metron-platform/metron-common/3rdPartyStellar.html Fri Apr 28 03:53:51 2017 @@ -0,0 +1,397 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-04-27 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170427" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Custom Stellar Functions</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Custom Stellar Functions</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-04-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-down"></i> + Platform</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-platform/metron-api/index.html" title="Api"> + <i class="none"></i> + Api</a> + </li> + + <li> + + <a href="../../metron-platform/metron-common/index.html" title="Common"> + <i class="icon-chevron-down"></i> + Common</a> + <ul class="nav nav-list"> + + <li class="active"> + + <a href="#"><i class="none"></i>3rdPartyStellar</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-platform/metron-data-management/index.html" title="Data-management"> + <i class="none"></i> + Data-management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"> + <i class="none"></i> + Enrichment</a> + </li> + + <li> + + <a href="../../metron-platform/metron-indexing/index.html" title="Indexing"> + <i class="none"></i> + Indexing</a> + </li> + + <li> + + <a href="../../metron-platform/metron-management/index.html" title="Management"> + <i class="none"></i> + Management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> + <i class="none"></i> + Parsers</a> + </li> + + <li> + + <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"> + <i class="none"></i> + Pcap-backend</a> + </li> + + <li> + + <a href="../../metron-platform/metron-writer/index.html" title="Writer"> + <i class="none"></i> + Writer</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Custom Stellar Functions</h1> +<p><a name="Custom_Stellar_Functions"></a></p> +<p>Metron is fundamentally a programmable, extensible system and Stellar is the extension language. We have some great Stellar functions available out of the box and we’ll be adding more over time, but they may not quite scratch quite your particular itch. </p> +<p>Of course, we’d love to have your contribution inside of Metron if you think it general purpose enough, but not every function is general-purpose or it may rely on libraries those licenses aren’t acceptable for an Apache project. In that case, then you will be wondering how to add your custom function to a running instance of Metron.</p> +<div class="section"> +<h2><a name="Building_Your_Own_Function"></a>Building Your Own Function</h2> +<p>Let’s say that I need a function that returns the current time in milliseconds since the epoch. I notice that there’s nothing like that currently in Metron, so I embark on the adventure of adding it for my cluster.</p> +<p>I will presume that you have an installed Metron into your local maven repo via <tt>mvn install</tt> . In the future, when we publish to a maven repo, you will not need this. I will depend on 0.4.0 for the purpose of this demonstration</p> +<div class="section"> +<h3><a name="Hack_Hack_Hack"></a>Hack, Hack, Hack</h3> +<p>I like to use Maven, so we’ll use that for this demonstration, but you can use whatever build system that you like. Here’s my favorite way to build a project with groupId <tt>com.mycompany.stellar</tt> and artifactId of <tt>tempus</tt> <tt>mvn archetype:create -DgroupId=com.mycompany.stellar -DartifactId=tempus -DarchetypeArtifactId=maven-archetype-quickstart</tt></p> +<p>First, we should depend on <tt>metron-common</tt> and we can do that by adjusting the <tt>pom.xml</tt> just created:</p> + +<div class="source"> +<div class="source"> +<pre><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> + <modelVersion>4.0.0</modelVersion> + + <groupId>com.mycompany.stellar</groupId> + <artifactId>tempus</artifactId> + <version>1.0-SNAPSHOT</version> + <packaging>jar</packaging> + + <name>Stellar Time Functions</name> + <url>http://mycompany.com</url> + + <properties> + <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> + </properties> + + <dependencies> + <dependency> + <groupId>org.apache.metron</groupId> + <artifactId>metron-common</artifactId> + <version>0.4.0</version> + <!-- NOTE: We will want to depend on the deployed common on the classpath. --> + <scope>provided</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <version>3.8.1</version> + <scope>test</scope> + </dependency> + </dependencies> +</project> +</pre></div></div> +<p>Let’s add our implementation in <tt>src/main/java/com/mycompany/stellar/TimeFunctions.java</tt> with the following content:</p> + +<div class="source"> +<div class="source"> +<pre>package com.notmetron.stellar; + +import org.apache.metron.common.dsl.Context; +import org.apache.metron.common.dsl.ParseException; +import org.apache.metron.common.dsl.Stellar; +import org.apache.metron.common.dsl.StellarFunction; + +import java.util.List; + +public class TimeFunction { + @Stellar( name="NOW", + description = "Right now!", + params = {}, + returns="Timestamp" + ) + public static class Now implements StellarFunction { + + public Object apply(List<Object> list, Context context) throws ParseException { + return System.currentTimeMillis(); + } + + public void initialize(Context context) { } + + public boolean isInitialized() { + return true; + } + } +} +</pre></div></div> +<p>Now we can build the project via <tt>mvn package</tt> which will create a <tt>target/tempus-1.0-SNAPSHOT.jar</tt> file.</p></div></div> +<div class="section"> +<h2><a name="Install_the_Function"></a>Install the Function</h2> +<p>Now that we have a jar with our custom function, we must make Metron aware of it.</p> +<div class="section"> +<h3><a name="Deploy_the_Jar"></a>Deploy the Jar</h3> +<p>First you need to place the jar in HDFS, if we have it on an access node, one way to do that is:</p> + +<ul> + +<li><tt>hadoop fs -put tempus-1.0-SNAPSHOT.jar /apps/metron/stellar</tt> This presumes that:</li> + +<li>you’ve standardized on <tt>/apps/metron/stellar</tt> as the location for custom jars</li> + +<li>you are running the command from an access node with the <tt>hadoop</tt> command installed</li> + +<li>you are running from a user that has write access to <tt>/apps/metron/stellar</tt></li> +</ul></div> +<div class="section"> +<h3><a name="Set_Global_Config"></a>Set Global Config</h3> +<p>You may not need this if your Metron administrator already has this setup.</p> +<p>With that dispensed with, we need to ensure that Metron knows to look at that location. We need to ensure that the <tt>stellar.function.paths</tt> property in the <tt>global.json</tt> is in place that makes Metron aware to look for Stellar functions in <tt>/apps/metron/stellar</tt> on HDFS. </p> +<p>This property looks like, the following for a vagrant install</p> + +<div class="source"> +<div class="source"> +<pre>{ + "es.clustername": "metron", + "es.ip": "node1", + "es.port": "9300", + "es.date.format": "yyyy.MM.dd.HH", + "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/.*.jar", +} +</pre></div></div> +<p>The <tt>stellar.function.paths</tt> property takes a comma separated list of URIs or URIs with regex expressions at the end. Also, note path is prefaced by the HDFS default name, which, if you do not know, can be found by executing, <tt>hdfs getconf -confKey fs.default.name</tt>, such as</p> + +<div class="source"> +<div class="source"> +<pre>[root@node1 ~]# hdfs getconf -confKey fs.default.name +hdfs://node1:8020 +</pre></div></div></div> +<div class="section"> +<h3><a name="Use_the_Function"></a>Use the Function</h3> +<p>Now that we have deployed the function, if we want to use it, any running topologies that use Stellar will need to be restarted.</p> +<p>Beyond that, let’s take a look at it in the REPL:</p> + +<div class="source"> +<div class="source"> +<pre>Stellar, Go! +Please note that functions are loading lazily in the background and will be unavailable until loaded fully. +{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH, stellar.function.paths=hdfs://node1:8020/apps/metron/stellar/.*.jar, profiler.client.period.duration=1, profiler.client.period.duration.units=MINUTES} +[Stellar]>>> # Get the help for NOW +[Stellar]>>> ?NOW +Functions loaded, you may refer to functions now... +NOW +Description: Right now! + +Returns: Timestamp +[Stellar]>>> # Try to run the NOW function, which we added: +[Stellar]>>> NOW() +1488400515655 +[Stellar]>>> # Looks like I got a timestamp, success! +</pre></div></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
