http://git-wip-us.apache.org/repos/asf/metron/blob/f7a94f2e/site/current-book/metron-platform/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/index.html b/site/current-book/metron-platform/index.html index 0546ef1..44c1804 100644 --- a/site/current-book/metron-platform/index.html +++ b/site/current-book/metron-platform/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-02-23 + | Generated by Apache Maven Doxia at 2017-06-27 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Current Build</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> @@ -30,14 +30,11 @@ <div class="container-fluid"> <div id="banner"> <div class="pull-left"> - <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> - <img src="../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> </a> </div> - <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> - <img src="../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> - </a> - </div> + <div class="pull-right"> </div> <div class="clear"><hr/></div> </div> @@ -51,8 +48,8 @@ </li> <li class="divider ">/</li> <li class=""> - <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> - Metron-Incubating</a> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> </li> <li class="divider ">/</li> <li class=""> @@ -64,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> </ul> </div> @@ -78,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../index.html" title="Metron"> @@ -99,7 +96,7 @@ <i class="icon-chevron-right"></i> Analytics</a> </li> - + <li> <a href="../metron-deployment/index.html" title="Deployment"> @@ -113,7 +110,21 @@ <i class="none"></i> Docker</a> </li> - + + <li> + + <a href="../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + <li class="active"> <a href="#"><i class="icon-chevron-down"></i>Platform</a> @@ -125,13 +136,13 @@ <i class="none"></i> Api</a> </li> - + <li> <a href="../metron-platform/metron-common/index.html" title="Common"> - <i class="none"></i> + <i class="icon-chevron-right"></i> Common</a> - </li> + </li> <li> @@ -174,9 +185,16 @@ <i class="none"></i> Pcap-backend</a> </li> + + <li> + + <a href="../metron-platform/metron-writer/index.html" title="Writer"> + <i class="none"></i> + Writer</a> + </li> </ul> </li> - + <li> <a href="../metron-sensors/index.html" title="Sensors"> @@ -220,11 +238,11 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --><h1>Current Build</h1> <p><a name="Current_Build"></a></p> -<p>The latest build of metron-platform is 0.3.1.</p> +<p>The latest build of metron-platform is 0.4.0.</p> <p>We are still in the process of merging/porting additional features from our production code base into this open source release. This release will be followed by a number of additional beta releases until the port is complete. We will also work on getting additional documentation and user/developer guides to the community as soon as we can. At this time we offer no support for the beta software, but will try to respond to requests as promptly as we can.</p> <p><a name="metron-platform"></a></p> <h1>metron-platform</h1> -<p>Extensible set of Storm topologies and topology attributes for streaming, enriching, indexing, and storing telemetry in Hadoop. General information on Metron is available at <a class="externalLink" href="https://metron.incubator.apache.org/";>https://metron.incubator.apache.org/</a></p> +<p>Extensible set of Storm topologies and topology attributes for streaming, enriching, indexing, and storing telemetry in Hadoop. General information on Metron is available at <a class="externalLink" href="https://metron.apache.org/";>https://metron.apache.org/</a></p> <p><a name="Documentation"></a></p> <h1>Documentation</h1> <p>Please see documentation within each individual module for description and usage instructions. Sample topologies are provided under Metron_Topologies to get you started with the framework. We pre-assume knowledge of Hadoop, Storm, Kafka, and HBase.</p> @@ -236,8 +254,9 @@ limitations under the License. --><h1>Current Build</h1> <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017. - All Rights Reserved. + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. </div>
http://git-wip-us.apache.org/repos/asf/metron/blob/f7a94f2e/site/current-book/metron-platform/metron-api/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-api/index.html b/site/current-book/metron-platform/metron-api/index.html index fe2b9a6..607e4aa 100644 --- a/site/current-book/metron-platform/metron-api/index.html +++ b/site/current-book/metron-platform/metron-api/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-02-23 + | Generated by Apache Maven Doxia at 2017-06-27 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Metron PCAP Service</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -30,14 +30,11 @@ <div class="container-fluid"> <div id="banner"> <div class="pull-left"> - <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> </a> </div> - <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> - <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> - </a> - </div> + <div class="pull-right"> </div> <div class="clear"><hr/></div> </div> @@ -51,8 +48,8 @@ </li> <li class="divider ">/</li> <li class=""> - <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> - Metron-Incubating</a> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> </li> <li class="divider ">/</li> <li class=""> @@ -64,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> </ul> </div> @@ -78,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -99,7 +96,7 @@ <i class="icon-chevron-right"></i> Analytics</a> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -113,7 +110,21 @@ <i class="none"></i> Docker</a> </li> - + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + <li> <a href="../../metron-platform/index.html" title="Platform"> @@ -125,13 +136,13 @@ <a href="#"><i class="none"></i>Api</a> </li> - + <li> <a href="../../metron-platform/metron-common/index.html" title="Common"> - <i class="none"></i> + <i class="icon-chevron-right"></i> Common</a> - </li> + </li> <li> @@ -174,9 +185,16 @@ <i class="none"></i> Pcap-backend</a> </li> + + <li> + + <a href="../../metron-platform/metron-writer/index.html" title="Writer"> + <i class="none"></i> + Writer</a> + </li> </ul> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> @@ -272,8 +290,9 @@ <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017. - All Rights Reserved. + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. </div> http://git-wip-us.apache.org/repos/asf/metron/blob/f7a94f2e/site/current-book/metron-platform/metron-common/3rdPartyStellar.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-common/3rdPartyStellar.html b/site/current-book/metron-platform/metron-common/3rdPartyStellar.html new file mode 100644 index 0000000..3e7b190 --- /dev/null +++ b/site/current-book/metron-platform/metron-common/3rdPartyStellar.html @@ -0,0 +1,398 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-06-27 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Custom Stellar Functions</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Custom Stellar Functions</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-down"></i> + Platform</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-platform/metron-api/index.html" title="Api"> + <i class="none"></i> + Api</a> + </li> + + <li> + + <a href="../../metron-platform/metron-common/index.html" title="Common"> + <i class="icon-chevron-down"></i> + Common</a> + <ul class="nav nav-list"> + + <li class="active"> + + <a href="#"><i class="none"></i>3rdPartyStellar</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-platform/metron-data-management/index.html" title="Data-management"> + <i class="none"></i> + Data-management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"> + <i class="none"></i> + Enrichment</a> + </li> + + <li> + + <a href="../../metron-platform/metron-indexing/index.html" title="Indexing"> + <i class="none"></i> + Indexing</a> + </li> + + <li> + + <a href="../../metron-platform/metron-management/index.html" title="Management"> + <i class="none"></i> + Management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> + <i class="none"></i> + Parsers</a> + </li> + + <li> + + <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"> + <i class="none"></i> + Pcap-backend</a> + </li> + + <li> + + <a href="../../metron-platform/metron-writer/index.html" title="Writer"> + <i class="none"></i> + Writer</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Custom Stellar Functions</h1> +<p><a name="Custom_Stellar_Functions"></a></p> +<p>Metron is fundamentally a programmable, extensible system and Stellar is the extension language. We have some great Stellar functions available out of the box and we’ll be adding more over time, but they may not quite scratch quite your particular itch. </p> +<p>Of course, we’d love to have your contribution inside of Metron if you think it general purpose enough, but not every function is general-purpose or it may rely on libraries those licenses aren’t acceptable for an Apache project. In that case, then you will be wondering how to add your custom function to a running instance of Metron.</p> +<div class="section"> +<h2><a name="Building_Your_Own_Function"></a>Building Your Own Function</h2> +<p>Let’s say that I need a function that returns the current time in milliseconds since the epoch. I notice that there’s nothing like that currently in Metron, so I embark on the adventure of adding it for my cluster.</p> +<p>I will presume that you have an installed Metron into your local maven repo via <tt>mvn install</tt> . In the future, when we publish to a maven repo, you will not need this. I will depend on 0.4.0 for the purpose of this demonstration</p> +<div class="section"> +<h3><a name="Hack_Hack_Hack"></a>Hack, Hack, Hack</h3> +<p>I like to use Maven, so we’ll use that for this demonstration, but you can use whatever build system that you like. Here’s my favorite way to build a project with groupId <tt>com.mycompany.stellar</tt> and artifactId of <tt>tempus</tt> <tt>mvn archetype:create -DgroupId=com.mycompany.stellar -DartifactId=tempus -DarchetypeArtifactId=maven-archetype-quickstart</tt></p> +<p>First, we should depend on <tt>metron-common</tt> and we can do that by adjusting the <tt>pom.xml</tt> just created:</p> + +<div class="source"> +<div class="source"> +<pre><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> + <modelVersion>4.0.0</modelVersion> + + <groupId>com.mycompany.stellar</groupId> + <artifactId>tempus</artifactId> + <version>1.0-SNAPSHOT</version> + <packaging>jar</packaging> + + <name>Stellar Time Functions</name> + <url>http://mycompany.com</url> + + <properties> + <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> + </properties> + + <dependencies> + <dependency> + <groupId>org.apache.metron</groupId> + <artifactId>metron-common</artifactId> + <version>0.4.0</version> + <!-- NOTE: We will want to depend on the deployed common on the classpath. --> + <scope>provided</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <version>3.8.1</version> + <scope>test</scope> + </dependency> + </dependencies> +</project> +</pre></div></div> +<p>Let’s add our implementation in <tt>src/main/java/com/mycompany/stellar/TimeFunctions.java</tt> with the following content:</p> + +<div class="source"> +<div class="source"> +<pre>package com.notmetron.stellar; + +import org.apache.metron.common.dsl.Context; +import org.apache.metron.common.dsl.ParseException; +import org.apache.metron.common.dsl.Stellar; +import org.apache.metron.common.dsl.StellarFunction; + +import java.util.List; + +public class TimeFunction { + @Stellar( name="NOW", + description = "Right now!", + params = {}, + returns="Timestamp" + ) + public static class Now implements StellarFunction { + + public Object apply(List<Object> list, Context context) throws ParseException { + return System.currentTimeMillis(); + } + + public void initialize(Context context) { } + + public boolean isInitialized() { + return true; + } + } +} +</pre></div></div> +<p>Now we can build the project via <tt>mvn package</tt> which will create a <tt>target/tempus-1.0-SNAPSHOT.jar</tt> file.</p></div></div> +<div class="section"> +<h2><a name="Install_the_Function"></a>Install the Function</h2> +<p>Now that we have a jar with our custom function, we must make Metron aware of it.</p> +<div class="section"> +<h3><a name="Deploy_the_Jar"></a>Deploy the Jar</h3> +<p>First you need to place the jar in HDFS, if we have it on an access node, one way to do that is:</p> + +<ul> + +<li><tt>hadoop fs -put tempus-1.0-SNAPSHOT.jar /apps/metron/stellar</tt> This presumes that:</li> + +<li>you’ve standardized on <tt>/apps/metron/stellar</tt> as the location for custom jars</li> + +<li>you are running the command from an access node with the <tt>hadoop</tt> command installed</li> + +<li>you are running from a user that has write access to <tt>/apps/metron/stellar</tt></li> +</ul></div> +<div class="section"> +<h3><a name="Set_Global_Config"></a>Set Global Config</h3> +<p>You may not need this if your Metron administrator already has this setup.</p> +<p>With that dispensed with, we need to ensure that Metron knows to look at that location. We need to ensure that the <tt>stellar.function.paths</tt> property in the <tt>global.json</tt> is in place that makes Metron aware to look for Stellar functions in <tt>/apps/metron/stellar</tt> on HDFS. </p> +<p>This property looks like, the following for a vagrant install</p> + +<div class="source"> +<div class="source"> +<pre>{ + "es.clustername": "metron", + "es.ip": "node1", + "es.port": "9300", + "es.date.format": "yyyy.MM.dd.HH", + "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/.*.jar", +} +</pre></div></div> +<p>The <tt>stellar.function.paths</tt> property takes a comma separated list of URIs or URIs with regex expressions at the end. Also, note path is prefaced by the HDFS default name, which, if you do not know, can be found by executing, <tt>hdfs getconf -confKey fs.default.name</tt>, such as</p> + +<div class="source"> +<div class="source"> +<pre>[root@node1 ~]# hdfs getconf -confKey fs.default.name +hdfs://node1:8020 +</pre></div></div></div> +<div class="section"> +<h3><a name="Use_the_Function"></a>Use the Function</h3> +<p>Now that we have deployed the function, if we want to use it, any running topologies that use Stellar will need to be restarted.</p> +<p>Beyond that, let’s take a look at it in the REPL:</p> + +<div class="source"> +<div class="source"> +<pre>Stellar, Go! +Please note that functions are loading lazily in the background and will be unavailable until loaded fully. +{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH, stellar.function.paths=hdfs://node1:8020/apps/metron/stellar/.*.jar, profiler.client.period.duration=1, profiler.client.period.duration.units=MINUTES} +[Stellar]>>> # Get the help for NOW +[Stellar]>>> ?NOW +Functions loaded, you may refer to functions now... +NOW +Description: Right now! + +Returns: Timestamp +[Stellar]>>> # Try to run the NOW function, which we added: +[Stellar]>>> NOW() +1488400515655 +[Stellar]>>> # Looks like I got a timestamp, success! +</pre></div></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/metron/blob/f7a94f2e/site/current-book/metron-platform/metron-common/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-common/index.html b/site/current-book/metron-platform/metron-common/index.html index c116276..7ddbc27 100644 --- a/site/current-book/metron-platform/metron-common/index.html +++ b/site/current-book/metron-platform/metron-common/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-02-23 + | Generated by Apache Maven Doxia at 2017-06-27 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Contents</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -30,14 +30,11 @@ <div class="container-fluid"> <div id="banner"> <div class="pull-left"> - <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> </a> </div> - <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> - <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> - </a> - </div> + <div class="pull-right"> </div> <div class="clear"><hr/></div> </div> @@ -51,8 +48,8 @@ </li> <li class="divider ">/</li> <li class=""> - <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> - Metron-Incubating</a> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> </li> <li class="divider ">/</li> <li class=""> @@ -64,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> </ul> </div> @@ -78,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -99,7 +96,7 @@ <i class="icon-chevron-right"></i> Analytics</a> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -113,7 +110,21 @@ <i class="none"></i> Docker</a> </li> - + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + <li> <a href="../../metron-platform/index.html" title="Platform"> @@ -127,11 +138,20 @@ <i class="none"></i> Api</a> </li> - + <li class="active"> - <a href="#"><i class="none"></i>Common</a> - </li> + <a href="#"><i class="icon-chevron-down"></i>Common</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-platform/metron-common/3rdPartyStellar.html" title="3rdPartyStellar"> + <i class="none"></i> + 3rdPartyStellar</a> + </li> + </ul> + </li> <li> @@ -174,9 +194,16 @@ <i class="none"></i> Pcap-backend</a> </li> + + <li> + + <a href="../../metron-platform/metron-writer/index.html" title="Writer"> + <i class="none"></i> + Writer</a> + </li> </ul> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> @@ -218,6 +245,8 @@ <li><a href="#Stellar_Core_Functions">Stellar Core Functions</a></li> +<li><a href="#Stellar_Benchmarks">Stellar Benchmarks</a></li> + <li><a href="#Stellar_Shell">Stellar Shell</a></li> </ul></li> @@ -234,7 +263,25 @@ <li>Referencing fields in the enriched JSON</li> -<li>Simple boolean operations: <tt>and</tt>, <tt>not</tt>, <tt>or</tt></li> +<li>String literals are quoted with either <tt>'</tt> or <tt>"</tt>.</li> + +<li>String literals support escaping for <tt>'</tt>, <tt>"</tt>, <tt>\t</tt>, <tt>\r</tt>, <tt>\n</tt>, and backslash + +<ul> + +<li>The literal <tt>'\'foo\''</tt> would represent <tt>'foo'</tt></li> + +<li>The literal <tt>"\"foo\""</tt> would represent <tt>"foo"</tt></li> + +<li>The literal <tt>'foo \\ bar'</tt> would represent <tt>foo \ bar</tt></li> + </ul></li> + +<li>Simple boolean operations: <tt>and</tt>, <tt>not</tt>, <tt>or</tt> + +<ul> + +<li>Boolean expressions are short-circuited (e.g. <tt>true or FUNC()</tt> would never execute <tt>FUNC</tt>)</li> + </ul></li> <li>Simple arithmetic operations: <tt>*</tt>, <tt>/</tt>, <tt>+</tt>, <tt>-</tt> on real numbers or integers</li> @@ -250,7 +297,7 @@ <li>The ability to have parenthesis to make order of operations explicit</li> -<li>User defined functions</li> +<li>User defined functions, including Lambda expressions</li> </ul> <div class="section"> <h2><a name="Stellar_Language_Keywords"></a>Stellar Language Keywords</h2> @@ -390,6 +437,51 @@ </ol> <p>The <tt>!=</tt> operator is the negation of the above.</p></div> <div class="section"> +<h2><a name="Stellar_Language_Lambda_Expressions"></a>Stellar Language Lambda Expressions</h2> +<p>Stellar provides the capability to pass lambda expressions to functions which wish to support that layer of indirection. The syntax is:</p> + +<ul> + +<li><tt>(named_variables) -> stellar_expression</tt> : Lambda expression with named variables + +<ul> + +<li>For instance, the lambda expression which calls <tt>TO_UPPER</tt> on a named argument <tt>x</tt> could be be expressed as <tt>(x) -> TO_UPPER(x)</tt>.</li> + </ul></li> + +<li><tt>var -> stellar_expression</tt> : Lambda expression with a single named variable, <tt>var</tt> + +<ul> + +<li>For instance, the lambda expression which calls <tt>TO_UPPER</tt> on a named argument <tt>x</tt> could be expressed as <tt>x -> TO_UPPER(x)</tt>. Note, this is more succinct but equivalent to the example directly above.</li> + </ul></li> + +<li><tt>() -> stellar_expression</tt> : Lambda expression with no named variables. + +<ul> + +<li>If no named variables are needed, you may omit the named variable section. For instance, the lambda expression which returns a constant <tt>false</tt> would be <tt>() -> false</tt></li> + </ul></li> +</ul> +<p>where </p> + +<ul> + +<li><tt>named_variables</tt> is a comma separated list of variables to use in the Stellar expression</li> + +<li><tt>stellar_expression</tt> is an arbitrary stellar expression</li> +</ul> +<p>In the core language functions, we support basic functional programming primitives such as</p> + +<ul> + +<li><tt>MAP</tt> - Applies a lambda expression over a list of input. For instance <tt>MAP([ 'foo', 'bar'], (x) -> TO_UPPER(x) )</tt> returns <tt>[ 'FOO', 'BAR' ]</tt></li> + +<li><tt>FILTER</tt> - Filters a list by a predicate in the form of a lambda expression. For instance <tt>FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )</tt> returns <tt>[ 'foo' ]</tt></li> + +<li><tt>REDUCE</tt> - Applies a function over a list of input. For instance <tt>REDUCE([ 1, 2, 3], (sum, x) -> sum + x, 0 )</tt> returns <tt>6</tt></li> +</ul></div> +<div class="section"> <h2><a name="Stellar_Core_Functions"></a>Stellar Core Functions</h2> <table border="0" class="table table-striped"> @@ -409,29 +501,49 @@ <tr class="a"> -<td><a href="../../metron-analytics/metron-statistics/index.html#BIN"> <tt>BIN</tt></a> </td> +<td><a href="#APPEND_IF_MISSING"> <tt>APPEND_IF_MISSING</tt></a> </td> </tr> <tr class="b"> -<td><a href="#BLOOM_ADD"> <tt>BLOOM_ADD</tt></a> </td> +<td><a href="../../metron-analytics/metron-statistics/index.html#BIN"> <tt>BIN</tt></a> </td> </tr> <tr class="a"> -<td><a href="#BLOOM_EXISTS"> <tt>BLOOM_EXISTS</tt></a> </td> +<td><a href="#BLOOM_ADD"> <tt>BLOOM_ADD</tt></a> </td> </tr> <tr class="b"> -<td><a href="#BLOOM_INIT"> <tt>BLOOM_INIT</tt></a> </td> +<td><a href="#BLOOM_EXISTS"> <tt>BLOOM_EXISTS</tt></a> </td> </tr> <tr class="a"> +<td><a href="#BLOOM_INIT"> <tt>BLOOM_INIT</tt></a> </td> + </tr> + +<tr class="b"> + <td><a href="#BLOOM_MERGE"> <tt>BLOOM_MERGE</tt></a> </td> </tr> +<tr class="a"> + +<td><a href="#CHOP"> <tt>CHOP</tt></a> </td> + </tr> + +<tr class="b"> + +<td><a href="#CHOMP"> <tt>CHOMP</tt></a> </td> + </tr> + +<tr class="a"> + +<td><a href="#COUNT_MATCHES"> <tt>COUNT_MATCHES</tt></a> </td> + </tr> + <tr class="b"> <td><a href="#DAY_OF_MONTH"> <tt>DAY_OF_MONTH</tt></a> </td> @@ -489,119 +601,129 @@ <tr class="a"> -<td><a href="#FORMAT"> <tt>FORMAT</tt></a> </td> +<td><a href="#FILTER"> <tt>FILTER</tt></a> </td> </tr> <tr class="b"> -<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_CARDINALITY"> <tt>HLLP_CARDINALITY</tt></a> </td> +<td><a href="#FORMAT"> <tt>FORMAT</tt></a> </td> </tr> <tr class="a"> -<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_INIT"> <tt>HLLP_INIT</tt></a> </td> +<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_CARDINALITY"> <tt>HLLP_CARDINALITY</tt></a> </td> </tr> <tr class="b"> -<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_MERGE"> <tt>HLLP_MERGE</tt></a> </td> +<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_INIT"> <tt>HLLP_INIT</tt></a> </td> </tr> <tr class="a"> -<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_OFFER"> <tt>HLLP_OFFER</tt></a> </td> +<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_MERGE"> <tt>HLLP_MERGE</tt></a> </td> </tr> <tr class="b"> -<td><a href="#GEO_GET"> <tt>GEO_GET</tt></a> </td> +<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_OFFER"> <tt>HLLP_OFFER</tt></a> </td> </tr> <tr class="a"> -<td><a href="#GET"> <tt>GET</tt></a> </td> +<td><a href="#GEO_GET"> <tt>GEO_GET</tt></a> </td> </tr> <tr class="b"> -<td><a href="#GET_FIRST"> <tt>GET_FIRST</tt></a> </td> +<td><a href="#GET"> <tt>GET</tt></a> </td> </tr> <tr class="a"> -<td><a href="#GET_LAST"> <tt>GET_LAST</tt></a> </td> +<td><a href="#GET_FIRST"> <tt>GET_FIRST</tt></a> </td> </tr> <tr class="b"> -<td><a href="#IN_SUBNET"> <tt>IN_SUBNET</tt></a> </td> +<td><a href="#GET_LAST"> <tt>GET_LAST</tt></a> </td> </tr> <tr class="a"> -<td><a href="#IS_DATE"> <tt>IS_DATE</tt></a> </td> +<td><a href="#IN_SUBNET"> <tt>IN_SUBNET</tt></a> </td> </tr> <tr class="b"> -<td><a href="#IS_DOMAIN"> <tt>IS_DOMAIN</tt></a> </td> +<td><a href="#IS_DATE"> <tt>IS_DATE</tt></a> </td> </tr> <tr class="a"> -<td><a href="#IS_EMAIL"> <tt>IS_EMAIL</tt></a> </td> +<td><a href="#IS_DOMAIN"> <tt>IS_DOMAIN</tt></a> </td> </tr> <tr class="b"> -<td><a href="#IS_EMPTY"> <tt>IS_EMPTY</tt></a> </td> +<td><a href="#IS_EMAIL"> <tt>IS_EMAIL</tt></a> </td> </tr> <tr class="a"> -<td><a href="#IS_INTEGER"> <tt>IS_INTEGER</tt></a> </td> +<td><a href="#IS_EMPTY"> <tt>IS_EMPTY</tt></a> </td> </tr> <tr class="b"> -<td><a href="#IS_IP"> <tt>IS_IP</tt></a> </td> +<td><a href="#IS_INTEGER"> <tt>IS_INTEGER</tt></a> </td> </tr> <tr class="a"> -<td><a href="#IS_URL"> <tt>IS_URL</tt></a> </td> +<td><a href="#IS_IP"> <tt>IS_IP</tt></a> </td> </tr> <tr class="b"> -<td><a href="#JOIN"> <tt>JOIN</tt></a> </td> +<td><a href="#IS_URL"> <tt>IS_URL</tt></a> </td> </tr> <tr class="a"> -<td><a href="#KAFKA_GET"> <tt>KAFKA_GET</tt></a> </td> +<td><a href="#JOIN"> <tt>JOIN</tt></a> </td> </tr> <tr class="b"> -<td><a href="#KAFKA_PROPS"> <tt>KAFKA_PROPS</tt></a> </td> +<td><a href="#KAFKA_GET"> <tt>KAFKA_GET</tt></a> </td> </tr> <tr class="a"> -<td><a href="#KAFKA_PUT"> <tt>KAFKA_PUT</tt></a> </td> +<td><a href="#KAFKA_PROPS"> <tt>KAFKA_PROPS</tt></a> </td> </tr> <tr class="b"> -<td><a href="#KAFKA_TAIL"> <tt>KAFKA_TAIL</tt></a> </td> +<td><a href="#KAFKA_PUT"> <tt>KAFKA_PUT</tt></a> </td> </tr> <tr class="a"> +<td><a href="#KAFKA_TAIL"> <tt>KAFKA_TAIL</tt></a> </td> + </tr> + +<tr class="b"> + <td><a href="#LENGTH"> <tt>LENGTH</tt></a> </td> </tr> +<tr class="a"> + +<td><a href="#LIST_ADD"> <tt>LIST_ADD</tt></a> </td> + </tr> + <tr class="b"> <td><a href="#MAAS_GET_ENDPOINT"> <tt>MAAS_GET_ENDPOINT</tt></a> </td> @@ -614,14 +736,24 @@ <tr class="b"> -<td><a href="#MAP_EXISTS"> <tt>MAP_EXISTS</tt></a> </td> +<td><a href="#MAP"> <tt>MAP</tt></a> </td> </tr> <tr class="a"> +<td><a href="#MAP_EXISTS"> <tt>MAP_EXISTS</tt></a> </td> + </tr> + +<tr class="b"> + <td><a href="#MONTH"> <tt>MONTH</tt></a> </td> </tr> +<tr class="a"> + +<td><a href="#PREPEND_IF_MISSING"> <tt>PREPEND_IF_MISSING</tt></a> </td> + </tr> + <tr class="b"> <td><a href="#PROFILE_GET"> <tt>PROFILE_GET</tt></a> </td> @@ -634,9 +766,19 @@ <tr class="b"> +<td><a href="#PROFILE_WINDOW"> <tt>PROFILE_WINDOW</tt></a> </td> + </tr> + +<tr class="a"> + <td><a href="#PROTOCOL_TO_NAME"> <tt>PROTOCOL_TO_NAME</tt></a> </td> </tr> +<tr class="b"> + +<td><a href="#REDUCE"> <tt>REDUCE</tt></a> </td> + </tr> + <tr class="a"> <td><a href="#REGEXP_MATCH"> <tt>REGEXP_MATCH</tt></a> </td> @@ -844,6 +986,26 @@ </tbody> </table> <div class="section"> +<h3><a name="APPEND_IF_MISSING"></a><tt>APPEND_IF_MISSING</tt></h3> + +<ul> + +<li>Description: Appends the suffix to the end of the string if the string does not already end with any of the suffixes.</li> + +<li>Input: + +<ul> + +<li>string - The string to be appended.</li> + +<li>suffix - The string suffix to append to the end of the string.</li> + +<li>additionalsuffix - Optional - Additional string suffix that is a valid terminator.</li> + </ul></li> + +<li>Returns: A new String if prefix was prepended, the same string otherwise.</li> +</ul></div> +<div class="section"> <h3><a name="BLOOM_ADD"></a><tt>BLOOM_ADD</tt></h3> <ul> @@ -914,6 +1076,56 @@ <li>Returns: Bloom Filter or null if the list is empty</li> </ul></div> <div class="section"> +<h3><a name="CHOP"></a><tt>CHOP</tt></h3> + +<ul> + +<li>Description: Remove the last character from a String</li> + +<li>Input: + +<ul> + +<li>string - the String to chop last character from, may be null</li> + </ul></li> + +<li>Returns: String without last character, null if null String input</li> +</ul></div> +<div class="section"> +<h3><a name="CHOMP"></a><tt>CHOMP</tt></h3> + +<ul> + +<li>Description: Removes one newline from end of a String if it’s there, otherwise leave it alone. A newline is “\n”, “\r”, or “\r\n”</li> + +<li>Input: + +<ul> + +<li>string - the String to chomp a newline from, may be null</li> + </ul></li> + +<li>Returns: String without newline, null if null String input</li> +</ul></div> +<div class="section"> +<h3><a name="COUNT_MATCHES"></a><tt>COUNT_MATCHES</tt></h3> + +<ul> + +<li>Description: Counts how many times the substring appears in the larger string.</li> + +<li>Input: + +<ul> + +<li>string - the CharSequence to check, may be null.</li> + +<li>substring/character - the substring or character to count, may be null.</li> + </ul></li> + +<li>Returns: the number of non-overlapping occurrences, 0 if either CharSequence is null.</li> +</ul></div> +<div class="section"> <h3><a name="DAY_OF_MONTH"></a><tt>DAY_OF_MONTH</tt></h3> <ul> @@ -1112,6 +1324,24 @@ <li>Returns: Last element of the list</li> </ul></div> <div class="section"> +<h3><a name="FILTER"></a><tt>FILTER</tt></h3> + +<ul> + +<li>Description: Applies a filter in the form of a lambda expression to a list. e.g. <tt>FILTER( [ 'foo', 'bar' ] , (x) -> x == 'foo')</tt> would yield <tt>[ 'foo']</tt></li> + +<li>Input: + +<ul> + +<li>list - List of arguments.</li> + +<li>predicate - The lambda expression to apply. This expression is assumed to take one argument and return a boolean.</li> + </ul></li> + +<li>Returns: The input list filtered by the predicate.</li> +</ul></div> +<div class="section"> <h3><a name="FORMAT"></a><tt>FORMAT</tt></h3> <ul> @@ -1442,6 +1672,24 @@ <li>Returns: Integer</li> </ul></div> <div class="section"> +<h3><a name="LIST_ADD"></a><tt>LIST_ADD</tt></h3> + +<ul> + +<li>Description: Adds an element to a list.</li> + +<li>Input: + +<ul> + +<li>list - List to add element to.</li> + +<li>element - Element to add to list</li> + </ul></li> + +<li>Returns: Resulting list with the item added at the end.</li> +</ul></div> +<div class="section"> <h3><a name="MAAS_GET_ENDPOINT"></a><tt>MAAS_GET_ENDPOINT</tt></h3> <ul> @@ -1480,6 +1728,24 @@ <li>Returns: The output of the model deployed as a REST endpoint in Map form. Assumes REST endpoint returns a JSON Map.</li> </ul></div> <div class="section"> +<h3><a name="MAP"></a><tt>MAP</tt></h3> + +<ul> + +<li>Description: Applies lambda expression to a list of arguments. e.g. <tt>MAP( [ 'foo', 'bar' ] , (x) -> TO_UPPER(x) )</tt> would yield <tt>[ 'FOO', 'BAR' ]</tt></li> + +<li>Input: + +<ul> + +<li>list - List of arguments.</li> + +<li>transform_expression - The lambda expression to apply. This expression is assumed to take one argument.</li> + </ul></li> + +<li>Returns: The input list transformed item-wise by the lambda expression.</li> +</ul></div> +<div class="section"> <h3><a name="MAP_EXISTS"></a><tt>MAP_EXISTS</tt></h3> <ul> @@ -1534,6 +1800,26 @@ <li>Returns: The current month (0-based).</li> </ul></div> <div class="section"> +<h3><a name="PREPEND_IF_MISSING"></a><tt>PREPEND_IF_MISSING</tt></h3> + +<ul> + +<li>Description: Prepends the prefix to the start of the string if the string does not already start with any of the prefixes.</li> + +<li>Input: + +<ul> + +<li>string - The string to be prepended.</li> + +<li>prefix - The string prefix to prepend to the start of the string.</li> + +<li>additionalprefix - Optional - Additional string prefix that is valid.</li> + </ul></li> + +<li>Returns: A new String if prefix was prepended, the same string otherwise.</li> +</ul></div> +<div class="section"> <h3><a name="PROFILE_GET"></a><tt>PROFILE_GET</tt></h3> <ul> @@ -1578,6 +1864,26 @@ <li>Returns: The selected profile measurement timestamps. These are ProfilePeriod objects.</li> </ul></div> <div class="section"> +<h3><a name="PROFILE_WINDOW"></a><tt>PROFILE_WINDOW</tt></h3> + +<ul> + +<li>Description: The profiler periods associated with a window selector statement from an optional reference timestamp.</li> + +<li>Input: + +<ul> + +<li>windowSelector - The statement specifying the window to select.</li> + +<li>now - Optional - The timestamp to use for now.</li> + +<li>config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter of the same name. Default is the empty Map, meaning no overrides.</li> + </ul></li> + +<li>Returns: The selected profile measurement periods. These are ProfilePeriod objects.</li> +</ul></div> +<div class="section"> <h3><a name="PROTOCOL_TO_NAME"></a><tt>PROTOCOL_TO_NAME</tt></h3> <ul> @@ -1594,6 +1900,26 @@ <li>Returns: The protocol name associated with the IANA number.</li> </ul></div> <div class="section"> +<h3><a name="REDUCE"></a><tt>REDUCE</tt></h3> + +<ul> + +<li>Description: Reduces a list by a binary lambda expression. That is, the expression takes two arguments. Usage example: <tt>REDUCE( [ 1, 2, 3 ] , (x, y) -> x + y, 0)</tt> would sum the input list, yielding <tt>6</tt>.</li> + +<li>Input: + +<ul> + +<li>list - List of arguments.</li> + +<li>binary_operation - The lambda expression function to apply to reduce the list. It is assumed that this takes two arguments, the first being the running total and the second being an item from the list.</li> + +<li>initial_value - The initial value to use.</li> + </ul></li> + +<li>Returns: The reduction of the list.</li> +</ul></div> +<div class="section"> <h3><a name="REGEXP_MATCH"></a><tt>REGEXP_MATCH</tt></h3> <ul> @@ -1979,8 +2305,71 @@ <li>The value in <tt>dc2tz</tt> associated with the value associated with field <tt>dc</tt>, defaulting to <tt>UTC</tt></li> </ul></div></div> <div class="section"> +<h2><a name="Stellar_Benchmarks"></a>Stellar Benchmarks</h2> +<p>A microbenchmarking utility is included to assist in executing microbenchmarks for Stellar functions. The utility can be executed via maven using the <tt>exec</tt> plugin, like so, from the <tt>metron-common</tt> directory:</p> + +<div class="source"> +<div class="source"> +<pre>mvn -DskipTests clean package && \ +mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.benchmark.StellarMicrobenchmark" -Dexec.args="..." +</pre></div></div> +<p>where <tt>exec.args</tt> can be one of the following:</p> + +<div class="source"> +<div class="source"> +<pre> -e,--expressions <FILE> Stellar expressions + -h,--help Generate Help screen + -n,--num_times <NUM> Number of times to run per expression (after + warmup). Default: 1000 + -o,--output <FILE> File to write output. + -p,--percentiles <NUM> Percentiles to calculate per run. Default: + 50.0,75.0,95.0,99.0 + -v,--variables <FILE> File containing a JSON Map of variables to use + -w,--warmup <NUM> Number of times for warmup per expression. + Default: 100 +</pre></div></div> +<p>For instance, to run with a set of Stellar expression in file <tt>/tmp/expressions.txt</tt>:</p> + +<div class="source"> +<div class="source"> +<pre> # simple functions + TO_UPPER('casey') + TO_LOWER(name) + # math functions + 1 + 2*(3 + int_num) / 10.0 + 1.5 + 2*(3 + double_num) / 10.0 + # conditionals + if ('foo' in ['foo']) OR one == very_nearly_one then 'one' else 'two' + 1 + 2*(3 + int_num) / 10.0 + #Network funcs + DOMAIN_TO_TLD(domain) + DOMAIN_REMOVE_SUBDOMAINS(domain) +</pre></div></div> +<p>And variables in file <tt>/tmp/variables.json</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "name" : "casey", + "int_num" : 1, + "double_num" : 17.5, + "one" : 1, + "very_nearly_one" : 1.000001, + "domain" : "www.google.com" +} +</pre></div></div> +<p>Written to file <tt>/tmp/output.txt</tt> would be the following command:</p> + +<div class="source"> +<div class="source"> +<pre>mvn -DskipTests clean package && \ +mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.benchmark.StellarMicrobenchmark" \ +-Dexec.args="-e /tmp/expressions.txt -v /tmp/variables.json -o ./output.json" +</pre></div></div></div> +<div class="section"> <h2><a name="Stellar_Shell"></a>Stellar Shell</h2> -<p>A REPL (Read Eval Print Loop) for the Stellar language that helps in debugging, troubleshooting and learning Stellar. The Stellar DSL (domain specific language) is used to act upon streaming data within Apache Storm. It is difficult to troubleshoot Stellar when it can only be executed within a Storm topology. This REPL is intended to help mitigate that problem by allowing a user to replicate data encountered in production, isolate initialization errors, or understand function resolution problems.</p> +<p>The Stellar Shell is a REPL (Read Eval Print Loop) for the Stellar language that helps troubleshooting, learning Stellar or even interacting with a live Metron cluster. </p> +<p>The Stellar DSL (domain specific language) is used to act upon streaming data within Apache Storm. It is difficult to troubleshoot Stellar when it can only be executed within a Storm topology. This REPL is intended to help mitigate that problem by allowing a user to replicate data encountered in production, isolate initialization errors, or understand function resolution problems.</p> <p>The shell supports customization via <tt>~/.inputrc</tt> as it is backed by a proper readline implementation. </p> <p>Shell-like operations are supported such as </p> @@ -1997,8 +2386,10 @@ <li>emacs or vi keybindings for edit mode</li> </ul> +<p>Note: Stellar classpath configuration from the global config is honored here if the REPL knows about zookeeper.</p> <div class="section"> <h3><a name="Getting_Started"></a>Getting Started</h3> +<p>To run the Stellar Shell from within a deployed Metron cluster, run the following command on the host where Metron is installed.</p> <div class="source"> <div class="source"> @@ -2042,7 +2433,7 @@ usage: stellar <div class="section"> <h4><a name="a-z_--zookeeper"></a><tt>-z, --zookeeper</tt></h4> <p><i>Optional</i></p> -<p>Attempts to connect to Zookeeper and read the Metron global configuration. Stellar functions may require the global configuration to work properly. If found, the global configuration values are printed to the console.</p> +<p>Attempts to connect to Zookeeper and read the Metron global configuration. Stellar functions may require the global configuration to work properly. If found, the global configuration values are printed to the console. If specified, then the classpath may be augmented by the paths specified in the stellar config in the global config.</p> <div class="source"> <div class="source"> @@ -2115,6 +2506,37 @@ IS_EMAIL args: address - The String to test ret: True if the string is a valid email address and false otherwise. [Stellar]>>> +</pre></div></div></div></div> +<div class="section"> +<h3><a name="Advanced_Usage"></a>Advanced Usage</h3> +<p>To run the Stellar Shell directly from the Metron source code, run a command like the following. Ensure that Metron has already been built and installed with <tt>mvn clean install -DskipTests</tt>.</p> + +<div class="source"> +<div class="source"> +<pre>$ mvn exec:java \ + -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \ + -pl metron-platform/metron-enrichment +... +Stellar, Go! +Please note that functions are loading lazily in the background and will be unavailable until loaded fully. +[Stellar]>>> Functions loaded, you may refer to functions now... +[Stellar]>>> %functions +ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GEO_GET, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_ SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR +</pre></div></div> +<p>Changing the project passed to the <tt>-pl</tt> argument will define which dependencies are included and ultimately which Stellar functions are available within the shell environment. </p> +<p>This can be useful for troubleshooting function resolution problems. The previous example defines which functions are available during Enrichment. For example, to determine which functions are available within the Profiler run the following.</p> + +<div class="source"> +<div class="source"> +<pre> $ mvn exec:java \ + -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \ + -pl metron-analytics/metron-profiler +... +Stellar, Go! +Please note that functions are loading lazily in the background and will be unavailable until loaded fully. +[Stellar]>>> Functions loaded, you may refer to functions now... +%functions +ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYS TEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR </pre></div></div> <p><a name="Global_Configuration"></a></p> <h1>Global Configuration</h1> @@ -2128,6 +2550,7 @@ IS_EMAIL "es.ip": "node1", "es.port": "9300", "es.date.format": "yyyy.MM.dd.HH", + "parser.error.topic": "indexing" "fieldValidations" : [ { "input" : [ "ip_src_addr", "ip_dst_addr" ], @@ -2138,9 +2561,70 @@ IS_EMAIL } ] } -</pre></div></div></div></div></div> +</pre></div></div> +<p><a name="Stellar_Configuration"></a></p> +<h1>Stellar Configuration</h1> +<p>Stellar can be configured in a variety of ways from the global config. In particular, there are three main configuration parameters around configuring Stellar:</p> + +<ul> + +<li><tt>stellar.function.paths</tt></li> + +<li><tt>stellar.function.resolver.includes</tt></li> + +<li><tt>stellar.function.resolver.excludes</tt></li> +</ul></div></div> +<div class="section"> +<h2><a name="stellar.function.paths"></a><tt>stellar.function.paths</tt></h2> +<p>If specified, Stellar will use a custom classloader which will wrap the context classloader and allow for the resolution of classes stored in jars not shipped with Metron and stored in a variety of mediums:</p> + +<ul> + +<li>On HDFS</li> + +<li>In tar.gz files</li> + +<li>At http/s locations</li> + +<li>At ftp locations</li> +</ul> +<p>This path is a comma separated list of </p> + +<ul> + +<li>URIs</li> + +<li>URIs with a regex pattern ending it for matching within a directory</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>{ + ... + "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/metron-management-0.4.0.jar, hdfs://node1:8020/apps/metron/3rdparty/.*.jar" +} +</pre></div></div> +<p>Please be aware that this classloader does not reload functions dynamically and the classpath specified here in the global config is read on topology start. A change in classpath, to be picked up, would necessitate a topology restart at the moment</p></div> <div class="section"> -<h2><a name="Validation_Framework"></a>Validation Framework</h2> +<h2><a name="stellar.function.resolver.includesexcludes"></a><tt>stellar.function.resolver.{includes,excludes}</tt></h2> +<p>If specified, this defines one or more regular expressions applied to the classes implementing the Stellar function that specify what should be included when searching for Stellar functions.</p> + +<ul> + +<li><tt>stellar.function.resolver.includes</tt> defines the list of classes to include.</li> + +<li><tt>stellar.function.resolver.excludes</tt> defines the list of classes to exclude.</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>{ + ... + "stellar.function.resolver.includes" : "org.apache.metron.*,com.myorg.stellar.*" +} +</pre></div></div> +<p><a name="Validation_Framework"></a></p> +<h1>Validation Framework</h1> <p>Inside of the global configuration, there is a validation framework in place that enables the validation that messages coming from all parsers are valid. This is done in the form of validation plugins where assertions about fields or whole messages can be made. </p> <p>The format for this is a <tt>fieldValidations</tt> field inside of global config. This is associated with an array of field validation objects structured like so:</p> @@ -2211,7 +2695,41 @@ IS_EMAIL <li>To push the configs into zookeeper on the singlenode vagrant machine: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i $METRON_HOME/config/zookeeper</tt></li> <li>To pull the configs from zookeeper to the singlenode vagrant machine disk: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PULL -o $METRON_HOME/config/zookeeper -f</tt></li> -</ul></div> +</ul> +<p><a name="Topology_Errors"></a></p> +<h1>Topology Errors</h1> +<p>Errors generated in Metron topologies are transformed into JSON format and follow this structure:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "exception": "java.lang.IllegalStateException: Unable to parse Message: ...", + "failed_sensor_type": "bro", + "stack": "java.lang.IllegalStateException: Unable to parse Message: ...", + "hostname": "node1", + "source:type": "error", + "raw_message": "{\"http\": {\"ts\":1488809627.000000.31915,\"uid\":\"C9JpSd2vFAWo3mXKz1\", ...", + "error_hash": "f7baf053f2d3c801a01d196f40f3468e87eea81788b2567423030100865c5061", + "error_type": "parser_error", + "message": "Unable to parse Message: {\"http\": {\"ts\":1488809627.000000.31915,\"uid\":\"C9JpSd2vFAWo3mXKz1\", ...", + "timestamp": 1488809630698 +} +</pre></div></div> +<p>Each topology can be configured to send error messages to a specific Kafka topic. The parser topologies retrieve this setting from the the <tt>parser.error.topic</tt> setting in the global config:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "es.clustername": "metron", + "es.ip": "node1", + "es.port": "9300", + "es.date.format": "yyyy.MM.dd.HH", + "parser.error.topic": "indexing" +} +</pre></div></div> +<p>Error topics for enrichment and threat intel errors are passed into the enrichment topology as flux properties named <tt>enrichment.error.topic</tt> and <tt>threat.intel.error.topic</tt>. These properties can be found in <tt>$METRON_HOME/config/enrichment.properties</tt>.</p> +<p>The error topic for indexing errors is passed into the indexing topology as a flux property named <tt>index.error.topic</tt>. This property can be found in either <tt>$METRON_HOME/config/elasticsearch.properties</tt> or <tt>$METRON_HOME/config/solr.properties</tt> depending on the search engine selected.</p> +<p>By default all error messages are sent to the <tt>indexing</tt> topic so that they are indexed and archived, just like other messages. The indexing config for error messages can be found at <tt>$METRON_HOME/config/zookeeper/indexing/error.json</tt>.</p></div> </div> </div> </div> @@ -2220,8 +2738,9 @@ IS_EMAIL <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017. - All Rights Reserved. + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. </div>
