http://git-wip-us.apache.org/repos/asf/metron/blob/79b38ea2/current-book/metron-sensors/bro-plugin-kafka/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-sensors/bro-plugin-kafka/index.html b/current-book/metron-sensors/bro-plugin-kafka/index.html index 3456bc7..638fdcd 100644 --- a/current-book/metron-sensors/bro-plugin-kafka/index.html +++ b/current-book/metron-sensors/bro-plugin-kafka/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Logging Bro Output to Kafka</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -75,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -131,7 +131,7 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../metron-platform/index.html" title="Platform"> @@ -166,6 +166,13 @@ </li> </ul> </li> + + <li> + + <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -173,7 +180,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../../use-cases/index.html" title="Use-cases"> @@ -371,7 +378,7 @@ event bro_init() &priority=-5 <li><tt>logs_to_send</tt> is mutually exclusive with <tt>$pred</tt>, thus for each log you want to set <tt>$pred</tt> on, you must individually setup a <tt>Log::add_filter</tt> and refrain from including that log in <tt>logs_to_send</tt>.</li> -<li>You can also filter IPv6 logs from within your Metron cluster <a href="../../metron-platform/metron-common/index.html#IS_IP">using Stellar</a>. In that case, you wouldn’t apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron.</li> +<li>You can also filter IPv6 logs from within your Metron cluster <a href="../../metron-stellar/stellar-common/index.html#IS_IP">using Stellar</a>. In that case, you wouldn’t apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron.</li> <li>It is also possible to use the <tt>is_v6_subnet()</tt> bro function in your predicate, as of their <a class="externalLink" href="https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5";>2.5 release</a>, however the above example should work on <a class="externalLink" href="https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4";>bro 2.4</a> and newer, which has been the focus of the kafka plugin.</li> </ul></div></div></div> @@ -498,7 +505,7 @@ redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "node <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/79b38ea2/current-book/metron-sensors/fastcapa/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-sensors/fastcapa/index.html b/current-book/metron-sensors/fastcapa/index.html index 38bae8a..59beb04 100644 --- a/current-book/metron-sensors/fastcapa/index.html +++ b/current-book/metron-sensors/fastcapa/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Fastcapa</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -131,14 +131,14 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> @@ -146,13 +146,6 @@ Sensors</a> <ul class="nav nav-list"> - <li> - - <a href="../../metron-sensors/bro-plugin-kafka/index.html" title="Bro-plugin-kafka"> - <i class="none"></i> - Bro-plugin-kafka</a> - </li> - <li class="active"> <a href="#"><i class="none"></i>Fastcapa</a> @@ -166,6 +159,13 @@ </li> </ul> </li> + + <li> + + <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -173,7 +173,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../../use-cases/index.html" title="Use-cases"> @@ -1087,7 +1087,7 @@ mount -a <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/metron/blob/79b38ea2/current-book/metron-sensors/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-sensors/index.html b/current-book/metron-sensors/index.html index f3ab985..8aff298 100644 --- a/current-book/metron-sensors/index.html +++ b/current-book/metron-sensors/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Metron Sensors</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../metron-deployment/index.html" title="Deployment"> @@ -131,14 +131,14 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li class="active"> <a href="#"><i class="icon-chevron-down"></i>Sensors</a> @@ -146,13 +146,6 @@ <li> - <a href="../metron-sensors/bro-plugin-kafka/index.html" title="Bro-plugin-kafka"> - <i class="none"></i> - Bro-plugin-kafka</a> - </li> - - <li> - <a href="../metron-sensors/fastcapa/index.html" title="Fastcapa"> <i class="none"></i> Fastcapa</a> @@ -166,6 +159,13 @@ </li> </ul> </li> + + <li> + + <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -173,7 +173,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../use-cases/index.html" title="Use-cases"> @@ -208,7 +208,7 @@ <ul> <li> -<p><a href="bro-plugin-kafka/index.html"><tt>bro-plugin-kafka</tt></a>: Provides integration between <a class="externalLink" href="https://www.bro.org/";>Bro</a> and Kafka. A Bro plugin that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others to process the data generated by Bro.</p></li> +<p><a class="externalLink" href="https://github.com/apache/metron-bro-plugin-kafka";><tt>metron-bro-plugin-kafka</tt></a>: Provides integration between <a class="externalLink" href="https://www.bro.org/";>Bro</a> and Kafka. A Bro plugin that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others to process the data generated by Bro.</p></li> <li> <p><a href="fastcapa/index.html"><tt>fastcapa</tt></a>: Performs fast network packet capture by leveraging Linux kernel-bypass and user space networking technology. The probe will bind to a network interface, capture network packets, and send the raw packet data to Kafka. This provides a scalable mechanism for ingesting high-volumes of network packet data.</p></li> @@ -224,7 +224,7 @@ <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/metron/blob/79b38ea2/current-book/metron-sensors/pycapa/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-sensors/pycapa/index.html b/current-book/metron-sensors/pycapa/index.html index 52d8f5a..8c46464 100644 --- a/current-book/metron-sensors/pycapa/index.html +++ b/current-book/metron-sensors/pycapa/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Pycapa</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -131,14 +131,14 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> @@ -148,13 +148,6 @@ <li> - <a href="../../metron-sensors/bro-plugin-kafka/index.html" title="Bro-plugin-kafka"> - <i class="none"></i> - Bro-plugin-kafka</a> - </li> - - <li> - <a href="../../metron-sensors/fastcapa/index.html" title="Fastcapa"> <i class="none"></i> Fastcapa</a> @@ -166,6 +159,13 @@ </li> </ul> </li> + + <li> + + <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -173,7 +173,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../../use-cases/index.html" title="Use-cases"> @@ -539,14 +539,19 @@ ${KAFKA_HOME}/bin/kafka-acls.sh \ INFO:root:'10' packet(s) in, '10' packet(s) out </pre></div></div></li> </ol> -<h1>FAQs</h1> -<p><b>Question</b>: How do I get more logs?</p> +<h1>FAQs</h1></div> +<div class="section"> +<h3><a name="How_do_I_get_more_logs"></a>How do I get more logs?</h3> <p>Use the following two command-line arguments to get detailed logging.</p> <div class="source"> <div class="source"> <pre>-X debug=all --log-level DEBUG -</pre></div></div></div></div> +</pre></div></div></div> +<div class="section"> +<h3><a name="When_I_run_Pycapa_against_a_Kafka_broker_with_Kerberos_enabled_why_do_I_get_an_error_like_No_such_configuration_property:_sasl.kerberos.principal"></a>When I run Pycapa against a Kafka broker with Kerberos enabled, why do I get an error like “No such configuration property: ‘sasl.kerberos.principal’”?</h3> +<p>This can be a confusing error message because <tt>sasl.kerberos.principal</tt> is indeed a valid property for librdkafka as defined <a class="externalLink" href="https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md";>here</a>. This is most likely because Pycapa is running against a version of Librdkafka without SASL support enabled. This might happen if you have accidentally installed multiple versions of Librdkafka and Pycapa is unexpectedly using the version without SASL support enabled.</p> +<p>Bottom Line: Make sure that Pycapa is running against a version of Librdkafka with SASL support enabled.</p></div></div> </div> </div> </div> @@ -555,7 +560,7 @@ ${KAFKA_HOME}/bin/kafka-acls.sh \ <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/metron/blob/79b38ea2/current-book/metron-stellar/stellar-3rd-party-example/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-stellar/stellar-3rd-party-example/index.html b/current-book/metron-stellar/stellar-3rd-party-example/index.html new file mode 100644 index 0000000..d60a128 --- /dev/null +++ b/current-book/metron-stellar/stellar-3rd-party-example/index.html @@ -0,0 +1,211 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2018-01-03 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Introduction</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Introduction</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> + <i class="none"></i> + Alerts</a> + </li> + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Stellar-3rd-party-example</a> + </li> + + <li> + + <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> + <i class="icon-chevron-right"></i> + Stellar-common</a> + </li> + + <li> + + <a href="../../use-cases/index.html" title="Use-cases"> + <i class="icon-chevron-right"></i> + Use-cases</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Introduction</h1> +<p><a name="Introduction"></a></p> +<p>This is just a sample 3rd party stellar function (which is used in our unit testing). If Stellar changes its API (e.g. the annotation changes classes), then this will need to be built and the output jar of this (in <tt>target</tt>) will need to be placed in <tt>../stellar-common/src/test/classpath-resources/</tt>.</p> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2018 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/metron/blob/79b38ea2/current-book/metron-stellar/stellar-common/3rdPartyStellar.html ---------------------------------------------------------------------- diff --git a/current-book/metron-stellar/stellar-common/3rdPartyStellar.html b/current-book/metron-stellar/stellar-common/3rdPartyStellar.html index d5633cc..eb9e054 100644 --- a/current-book/metron-stellar/stellar-common/3rdPartyStellar.html +++ b/current-book/metron-stellar/stellar-common/3rdPartyStellar.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Custom Stellar Functions</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -131,20 +131,27 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -159,7 +166,7 @@ </li> </ul> </li> - + <li> <a href="../../use-cases/index.html" title="Use-cases"> @@ -195,7 +202,7 @@ <div class="section"> <h2><a name="Building_Your_Own_Function"></a>Building Your Own Function</h2> <p>Let’s say that I need a function that returns the current time in milliseconds since the epoch. I notice that there’s nothing like that currently in Metron, so I embark on the adventure of adding it for my cluster.</p> -<p>I will presume that you have an installed Metron into your local maven repo via <tt>mvn install</tt> . In the future, when we publish to a maven repo, you will not need this. I will depend on 0.4.1 for the purpose of this demonstration</p> +<p>I will presume that you have an installed Metron into your local maven repo via <tt>mvn install</tt> . In the future, when we publish to a maven repo, you will not need this. I will depend on 0.4.2 for the purpose of this demonstration</p> <div class="section"> <h3><a name="Hack_Hack_Hack"></a>Hack, Hack, Hack</h3> <p>I like to use Maven, so we’ll use that for this demonstration, but you can use whatever build system that you like. Here’s my favorite way to build a project with groupId <tt>com.mycompany.stellar</tt> and artifactId of <tt>tempus</tt> <tt>mvn archetype:create -DgroupId=com.mycompany.stellar -DartifactId=tempus -DarchetypeArtifactId=maven-archetype-quickstart</tt></p> @@ -223,7 +230,7 @@ <dependency> <groupId>org.apache.metron</groupId> <artifactId>metron-common</artifactId> - <version>0.4.1</version> + <version>0.4.2</version> <!-- NOTE: We will want to depend on the deployed common on the classpath. --> <scope>provided</scope> </dependency> @@ -340,7 +347,7 @@ Returns: Timestamp <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/metron/blob/79b38ea2/current-book/metron-stellar/stellar-common/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-stellar/stellar-common/index.html b/current-book/metron-stellar/stellar-common/index.html index 933d2a9..0465dbb 100644 --- a/current-book/metron-stellar/stellar-common/index.html +++ b/current-book/metron-stellar/stellar-common/index.html @@ -1,15 +1,15 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> - <title>Metron – Contents</title> + <title>Metron – Stellar Language</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> <link rel="stylesheet" href="../../css/site.css" /> <link rel="stylesheet" href="../../css/print.css" media="print" /> @@ -57,12 +57,12 @@ Documentation</a> </li> <li class="divider ">/</li> - <li class="">Contents</li> + <li class="">Stellar Language</li> - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -131,20 +131,27 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li class="active"> @@ -159,7 +166,7 @@ </li> </ul> </li> - + <li> <a href="../../use-cases/index.html" title="Use-cases"> @@ -188,30 +195,38 @@ <div id="bodyColumn" class="span9" > - <h1>Contents</h1> -<p><a name="Contents"></a></p> + <h1>Stellar Language</h1> +<p><a name="Stellar_Language"></a></p> +<p>For a variety of components (threat intelligence triage and field transformations) we have the need to do simple computation and transformation using the data from messages as variables. For those purposes, there exists a simple, scaled down DSL created to do simple computation and transformation.</p> <ul> -<li><a href="#Stellar_Language">Stellar Language</a> +<li><a href="#Introduction">Introduction</a></li> + +<li><a href="#Stellar_Core_Functions">Stellar Core Functions</a></li> + +<li><a href="#Stellar_Benchmarks">Stellar Benchmarks</a></li> + +<li><a href="#Stellar_Shell">Stellar Shell</a> <ul> -<li><a href="#Stellar_Language_Keywords">Stellar Language Keywords</a></li> +<li><a href="#Getting_Started">Getting Started</a></li> -<li><a href="#Stellar_Core_Functions">Stellar Core Functions</a></li> +<li><a href="#Command_Line_Options">Command Line Options</a></li> -<li><a href="#Stellar_Benchmarks">Stellar Benchmarks</a></li> +<li><a href="#Variable_Assignment">Variable Assignment</a></li> -<li><a href="#Stellar_Shell">Stellar Shell</a></li> +<li><a href="#Magic_Commands">Magic Commands</a></li> + +<li><a href="#Advanced_Usage">Advanced Usage</a></li> </ul></li> <li><a href="#Stellar_Configuration">Stellar Configuration</a></li> </ul> -<p><a name="Stellar_Language"></a></p> -<h1>Stellar Language</h1> -<p>For a variety of components (threat intelligence triage and field transformations) we have the need to do simple computation and transformation using the data from messages as variables.<br />For those purposes, there exists a simple, scaled down DSL created to do simple computation and transformation.</p> -<p>The query language supports the following:</p> +<div class="section"> +<h2><a name="Introduction"></a>Introduction</h2> +<p>The Stellar language supports the following:</p> <ul> @@ -240,6 +255,8 @@ <li>if/then/else comparisons (i.e. <tt>if var1 < 10 then 'less than 10' else '10 or more'</tt>)</li> +<li>Simple match evaluations (i.e. <tt>match{ var1 < 10 => 'warn', var1 >= 10 => 'critical', default => 'info'}</tt></li> + <li>Determining whether a field exists (via <tt>exists</tt>)</li> <li>An <tt>in</tt> operator that works like the <tt>in</tt> in Python</li> @@ -249,7 +266,7 @@ <li>User defined functions, including Lambda expressions</li> </ul> <div class="section"> -<h2><a name="Stellar_Language_Keywords"></a>Stellar Language Keywords</h2> +<h3><a name="Stellar_Language_Keywords"></a>Stellar Language Keywords</h3> <p>The following keywords need to be single quote escaped in order to be used in Stellar expressions:</p> <table border="0" class="table table-striped"> @@ -291,41 +308,67 @@ <td align="center">in </td> -<td align="center">== </td> +<td align="center">NaN </td> -<td align="center">!= </td> +<td align="center">match </td> </tr> <tr class="b"> +<td align="center">default </td> + +<td align="center">== </td> + +<td align="center">!= </td> + <td align="center"><= </td> <td align="center">> </td> + </tr> + +<tr class="a"> <td align="center">>= </td> <td align="center">+ </td> <td align="center">- </td> - </tr> - -<tr class="a"> <td align="center">< </td> <td align="center">? </td> + </tr> + +<tr class="b"> <td align="center">* </td> <td align="center">/ </td> <td align="center">, </td> + +<td align="center">{ </td> + +<td align="center">} </td> + </tr> + +<tr class="a"> + +<td align="center">\=> </td> + +<td align="center"> </td> + +<td align="center"> </td> + +<td align="center"> </td> + +<td align="center"> </td> </tr> </tbody> </table> <p>Using parens such as: “foo” : “<ok>” requires escaping; “foo”: “'<ok>'”</p></div> <div class="section"> -<h2><a name="Stellar_Language_Inclusion_Checks_in_and_not_in"></a>Stellar Language Inclusion Checks (<tt>in</tt> and <tt>not in</tt>)</h2> +<h3><a name="Stellar_Language_Inclusion_Checks_in_and_not_in"></a>Stellar Language Inclusion Checks (<tt>in</tt> and <tt>not in</tt>)</h3> <ol style="list-style-type: decimal"> @@ -338,7 +381,7 @@ <li><tt>not in</tt> is the negation of the in expression. e.g. <tt>'grok' not in 'foobar' == true</tt></li> </ol></div> <div class="section"> -<h2><a name="Stellar_Language_Comparisons____"></a>Stellar Language Comparisons (<tt><</tt>, <tt><=</tt>, <tt>></tt>, <tt>>=</tt>)</h2> +<h3><a name="Stellar_Language_Comparisons____"></a>Stellar Language Comparisons (<tt><</tt>, <tt><=</tt>, <tt>></tt>, <tt>>=</tt>)</h3> <ol style="list-style-type: decimal"> @@ -362,7 +405,7 @@ <li>If none of the above are met then an exception is thrown.</li> </ol></div> <div class="section"> -<h2><a name="Stellar_Language_Equality_Check__"></a>Stellar Language Equality Check (<tt>==</tt>, <tt>!=</tt>)</h2> +<h3><a name="Stellar_Language_Equality_Check__"></a>Stellar Language Equality Check (<tt>==</tt>, <tt>!=</tt>)</h3> <p>Below is how the <tt>==</tt> operator is expected to work:</p> <ol style="list-style-type: decimal"> @@ -386,7 +429,7 @@ </ol> <p>The <tt>!=</tt> operator is the negation of the above.</p></div> <div class="section"> -<h2><a name="Stellar_Language_Lambda_Expressions"></a>Stellar Language Lambda Expressions</h2> +<h3><a name="Stellar_Language_Lambda_Expressions"></a>Stellar Language Lambda Expressions</h3> <p>Stellar provides the capability to pass lambda expressions to functions which wish to support that layer of indirection. The syntax is:</p> <ul> @@ -431,6 +474,35 @@ <li><tt>REDUCE</tt> - Applies a function over a list of input. For instance <tt>REDUCE([ 1, 2, 3], (sum, x) -> sum + x, 0 )</tt> returns <tt>6</tt></li> </ul></div> <div class="section"> +<h3><a name="Stellar_Language_Match_Expression"></a>Stellar Language Match Expression</h3> +<p>Stellar provides the capability to write match expressions, which are similar to switch statements commonly found in c like languages.</p> +<p>The syntax is:</p> + +<ul> + +<li><tt>match{ logical_expression1 => evaluation expression1, logical_expression2 => evaluation_expression2, default => default_expression}</tt></li> +</ul> +<p>Where:</p> + +<ul> + +<li><tt>logical_expression</tt> is a Stellar expression that evaluates to true or false. For instance <tt>var > 0</tt> or <tt>var > 0 AND var2 == 'foo'</tt> or <tt>IF ... THEN ... ELSE</tt></li> + +<li><tt>evaluation_expression</tt> is a Stellar Expression</li> + +<li><tt>default</tt> is a required default return value, should no logical expression match</li> +</ul> + +<blockquote> +<p>default is required </p> +<p>Lambda expressions are supported, but they must be no argument lambdas such as <tt>() -> STATEMENT</tt></p> +</blockquote> + +<ul> + +<li>Only the first clause that evaluates to true will be executed.</li> +</ul></div></div> +<div class="section"> <h2><a name="Stellar_Core_Functions"></a>Stellar Core Functions</h2> <table border="0" class="table table-striped"> @@ -585,17 +657,17 @@ <tr class="b"> -<td><a href="#FUZZY_LANGS"> <tt>FUZZY_LANGS</tt></a> </td> +<td><a href="#FORMAT"> <tt>FORMAT</tt></a> </td> </tr> <tr class="a"> -<td><a href="#FUZZY_SCORE"> <tt>FUZZY_SCORE</tt></a> </td> +<td><a href="#FUZZY_LANGS"> <tt>FUZZY_LANGS</tt></a> </td> </tr> <tr class="b"> -<td><a href="#FORMAT"> <tt>FORMAT</tt></a> </td> +<td><a href="#FUZZY_SCORE"> <tt>FUZZY_SCORE</tt></a> </td> </tr> <tr class="a"> @@ -660,22 +732,22 @@ <tr class="a"> -<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_CARDINALITY"> <tt>HLLP_CARDINALITY</tt></a> </td> +<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_ADD"> <tt>HLLP_ADD</tt></a> </td> </tr> <tr class="b"> -<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_INIT"> <tt>HLLP_INIT</tt></a> </td> +<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_CARDINALITY"> <tt>HLLP_CARDINALITY</tt></a> </td> </tr> <tr class="a"> -<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_MERGE"> <tt>HLLP_MERGE</tt></a> </td> +<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_INIT"> <tt>HLLP_INIT</tt></a> </td> </tr> <tr class="b"> -<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_OFFER"> <tt>HLLP_OFFER</tt></a> </td> +<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_MERGE"> <tt>HLLP_MERGE</tt></a> </td> </tr> <tr class="a"> @@ -720,79 +792,99 @@ <tr class="a"> -<td><a href="#IS_URL"> <tt>IS_URL</tt></a> </td> +<td><a href="#IS_NAN"> <tt>IS_NAN</tt></a> </td> </tr> <tr class="b"> -<td><a href="#JOIN"> <tt>JOIN</tt></a> </td> +<td><a href="#IS_URL"> <tt>IS_URL</tt></a> </td> </tr> <tr class="a"> -<td><a href="#KAFKA_GET"> <tt>KAFKA_GET</tt></a> </td> +<td><a href="#JOIN"> <tt>JOIN</tt></a> </td> </tr> <tr class="b"> -<td><a href="#KAFKA_PROPS"> <tt>KAFKA_PROPS</tt></a> </td> +<td><a href="#KAFKA_GET"> <tt>KAFKA_GET</tt></a> </td> </tr> <tr class="a"> -<td><a href="#KAFKA_PUT"> <tt>KAFKA_PUT</tt></a> </td> +<td><a href="#KAFKA_PROPS"> <tt>KAFKA_PROPS</tt></a> </td> </tr> <tr class="b"> -<td><a href="#KAFKA_TAIL"> <tt>KAFKA_TAIL</tt></a> </td> +<td><a href="#KAFKA_PUT"> <tt>KAFKA_PUT</tt></a> </td> </tr> <tr class="a"> -<td><a href="#LENGTH"> <tt>LENGTH</tt></a> </td> +<td><a href="#KAFKA_TAIL"> <tt>KAFKA_TAIL</tt></a> </td> </tr> <tr class="b"> -<td><a href="#LIST_ADD"> <tt>LIST_ADD</tt></a> </td> +<td><a href="#LENGTH"> <tt>LENGTH</tt></a> </td> </tr> <tr class="a"> -<td><a href="#LOG2"> <tt>LOG2</tt></a> </td> +<td><a href="#LIST_ADD"> <tt>LIST_ADD</tt></a> </td> </tr> <tr class="b"> -<td><a href="#LOG10"> <tt>LOG10</tt></a> </td> +<td><a href="#LOG2"> <tt>LOG2</tt></a> </td> </tr> <tr class="a"> -<td><a href="#LN"> <tt>LN</tt></a> </td> +<td><a href="#LOG10"> <tt>LOG10</tt></a> </td> </tr> <tr class="b"> -<td><a href="#MAAS_GET_ENDPOINT"> <tt>MAAS_GET_ENDPOINT</tt></a> </td> +<td><a href="#LN"> <tt>LN</tt></a> </td> </tr> <tr class="a"> -<td><a href="#MAAS_MODEL_APPLY"> <tt>MAAS_MODEL_APPLY</tt></a> </td> +<td><a href="#MAAS_GET_ENDPOINT"> <tt>MAAS_GET_ENDPOINT</tt></a> </td> </tr> <tr class="b"> -<td><a href="#MAP"> <tt>MAP</tt></a> </td> +<td><a href="#MAAS_MODEL_APPLY"> <tt>MAAS_MODEL_APPLY</tt></a> </td> </tr> <tr class="a"> +<td><a href="#MAP"> <tt>MAP</tt></a> </td> + </tr> + +<tr class="b"> + <td><a href="#MAP_EXISTS"> <tt>MAP_EXISTS</tt></a> </td> </tr> +<tr class="a"> + +<td><a href="#MAP_GET"> <tt>MAP_GET</tt></a> </td> + </tr> + +<tr class="b"> + +<td><a href="#MAX"> <tt>MAX</tt></a> </td> + </tr> + +<tr class="a"> + +<td><a href="#MIN"> <tt>MIN</tt></a> </td> + </tr> + <tr class="b"> <td><a href="#MONTH"> <tt>MONTH</tt></a> </td> @@ -890,12 +982,12 @@ <tr class="a"> -<td><a href="#SPLIT"> <tt>SPLIT</tt></a> </td> +<td><a href="#SIN"> <tt>SIN</tt></a> </td> </tr> <tr class="b"> -<td><a href="#SIN"> <tt>SIN</tt></a> </td> +<td><a href="#SPLIT"> <tt>SPLIT</tt></a> </td> </tr> <tr class="a"> @@ -1030,24 +1122,44 @@ <tr class="a"> -<td><a href="#TO_DOUBLE"> <tt>TO_DOUBLE</tt></a> </td> +<td><a href="#TLSH_DIST"> <tt>TLSH_DIST</tt></a> </td> </tr> <tr class="b"> -<td><a href="#TO_EPOCH_TIMESTAMP"> <tt>TO_EPOCH_TIMESTAMP</tt></a> </td> +<td><a href="#TO_DOUBLE"> <tt>TO_DOUBLE</tt></a> </td> </tr> <tr class="a"> -<td><a href="#TO_FLOAT"> <tt>TO_FLOAT</tt></a> </td> +<td><a href="#TO_EPOCH_TIMESTAMP"> <tt>TO_EPOCH_TIMESTAMP</tt></a> </td> </tr> <tr class="b"> +<td><a href="#TO_FLOAT"> <tt>TO_FLOAT</tt></a> </td> + </tr> + +<tr class="a"> + <td><a href="#TO_INTEGER"> <tt>TO_INTEGER</tt></a> </td> </tr> +<tr class="b"> + +<td><a href="#TO_JSON_LIST"> <tt>TO_JSON_LIST</tt></a> </td> + </tr> + +<tr class="a"> + +<td><a href="#TO_JSON_MAP"> <tt>TO_JSON_MAP</tt></a> </td> + </tr> + +<tr class="b"> + +<td><a href="#TO_JSON_OBJECT"> <tt>TO_JSON_OBJECT</tt></a> </td> + </tr> + <tr class="a"> <td><a href="#TO_LONG"> <tt>TO_LONG</tt></a> </td> @@ -1115,7 +1227,7 @@ <tr class="b"> -<td><a href="#ZIP_JAGGED"> <tt>ZIP_JAGGED</tt></a> </td> +<td><a href="#ZIP_LONGEST"> <tt>ZIP_LONGEST</tt></a> </td> </tr> </tbody> </table> @@ -1152,7 +1264,7 @@ <li>bloom - The bloom filter</li> -<li>value* - The values to add</li> +<li>value(s) - The value(s) to add</li> </ul></li> <li>Returns: Bloom Filter</li> @@ -1830,6 +1942,15 @@ <li>Returns: Last element of the list</li> </ul></div> <div class="section"> +<h3><a name="GET_SUPPORTED_ENCODINGS"></a><tt>GET_SUPPORTED_ENCODINGS</tt></h3> + +<ul> + +<li>Description: Returns a list of the encodings that are currently supported.</li> + +<li>Returns: A List of String</li> +</ul></div> +<div class="section"> <h3><a name="HASH"></a><tt>HASH</tt></h3> <ul> @@ -1843,9 +1964,34 @@ <li>toHash - value to hash.</li> <li>hashType - A valid string representation of a hashing algorithm. See ‘GET_HASHES_AVAILABLE’.</li> + +<li>config? - Configuration for the hash function in the form of a String to object map. + +<ul> + +<li>For forensic hash TLSH (see <a class="externalLink" href="https://github.com/trendmicro/tlsh";>https://github.com/trendmicro/tlsh</a> and Jonathan Oliver, Chun Cheng, and Yanggui Chen, TLSH - A Locality Sensitive Hash. 4th Cybercrime and Trustworthy Computing Workshop, Sydney, November 2013): + +<ul> + +<li>bucketSize : This defines the size of the hash created. Valid values are 128 (default) or 256 (the former results in a 70 character hash and latter results in 134 characters)</li> + +<li>checksumBytes : This defines how many bytes are used to capture the checksum. Valid values are 1 (default) and 3</li> + +<li>force : If true (the default) then a hash can be generated from as few as 50 bytes. If false, then at least 256 bytes are required. Insufficient variation or size in the bytes result in a null being returned.</li> + +<li>hashes : You can compute a second hash for use in fuzzy clustering TLSH signatures. The number of hashes is the lever to adjust the size of those clusters and "fuzzy" the clusters are. If this is specified, then one or more bins are created based on the specified size and the function will return a Map containing the bins.</li> + </ul></li> + +<li>For all other hashes: + +<ul> + +<li>charset : The character set to use (UTF8 is default).</li> + </ul></li> + </ul></li> </ul></li> -<li>Returns: A hex encoded string of a hashed value using the given algorithm. If ‘hashType’ is null then ‘00’, padded to the necessary length, will be returned. If ‘toHash’ is not able to be hashed or ‘hashType’ is null then null is returned.</li> +<li>Returns = A hex encoded string of a hashed value using the given algorithm. If ‘hashType’ is null then ‘00’, padded to the necessary length, will be returned. If ‘toHash’ is not able to be hashed or ‘hashType’ is null then null is returned.</li> </ul></div> <div class="section"> <h3><a name="IN_SUBNET"></a><tt>IN_SUBNET</tt></h3> @@ -1984,6 +2130,22 @@ <li>Returns: True if the string is an IP and false otherwise.</li> </ul></div> <div class="section"> +<h3><a name="IS_NAN"></a><tt>IS_NAN</tt></h3> + +<ul> + +<li>Description: Evaluates if the passed number is NaN. The number is evaluated as a double.</li> + +<li>Input: + +<ul> + +<li>number - number to evaluate"</li> + </ul></li> + +<li>Returns: True if the number is NaN, false if it is</li> +</ul></div> +<div class="section"> <h3><a name="IS_URL"></a><tt>IS_URL</tt></h3> <ul> @@ -2128,15 +2290,6 @@ <li>Returns: Resulting list with the item added at the end.</li> </ul></div> <div class="section"> -<h3><a name="GET_SUPPORTED_ENCODINGS"></a><tt>GET_SUPPORTED_ENCODINGS</tt></h3> - -<ul> - -<li>Description: Returns a list of the encodings that are currently supported.</li> - -<li>Returns: A List of String</li> -</ul></div> -<div class="section"> <h3><a name="LOG2"></a><tt>LOG2</tt></h3> <ul> @@ -2279,6 +2432,38 @@ <li>Returns: The object associated with the key in the map. If no value is associated with the key and default is specified, then default is returned. If no value is associated with the key or default, then null is returned.</li> </ul></div> <div class="section"> +<h3><a name="MAX"></a><tt>MAX</tt></h3> + +<ul> + +<li>Description: Returns the maximum value of a list of input values. + +<ul> + +<li>Input:</li> + +<li>"list - List of arguments. The list may only contain objects that are mutually comparable / ordinal (implement java.lang.Comparable interface). Multi type numeric comparisons are supported: MAX([10,15L,15.3]) would return 15.3, but MAX([‘23’,25]) will fail and return null as strings and numbers can’t be compared.</li> + </ul></li> + +<li>Returns: The maximum value of the list, or null if the list is empty or the input values were not comparable.</li> +</ul></div> +<div class="section"> +<h3><a name="MIN"></a><tt>MIN</tt></h3> + +<ul> + +<li>Description: Returns the minimum value of a list of input values. + +<ul> + +<li>Input:</li> + +<li>"list - List of arguments. The list may only contain objects that are mutually comparable / ordinal (implement java.lang.Comparable interface). Multi type numeric comparisons are supported: MIN([10,15L,15.3]) would return 10, but MIN([‘23’,25]) will fail and return null as strings and numbers can’t be compared.</li> + </ul></li> + +<li>Returns: The minimum value of the list, or null if the list is empty or the input values were not comparable.</li> +</ul></div> +<div class="section"> <h3><a name="MONTH"></a><tt>MONTH</tt></h3> <ul> @@ -2292,7 +2477,7 @@ <li>dateTime - The datetime as a long representing the milliseconds since unix epoch</li> </ul></li> -<li>Returns: The current month (0-based).</li> +<li>Returns: The current month (0-based)</li> </ul></div> <div class="section"> <h3><a name="MULTISET_ADD"></a><tt>MULTISET_ADD</tt></h3> @@ -2323,7 +2508,7 @@ <ul> -<li>input? - An initialization of the multiset</li> +<li>input (optional) - An initialization of the multiset</li> </ul></li> <li>Returns: A multiset</li> @@ -2581,7 +2766,7 @@ <ul> -<li>input? - An initialization of the set</li> +<li>input (optional) - An initialization of the set</li> </ul></li> <li>Returns: A Set</li> @@ -2637,54 +2822,54 @@ <li>Returns: The sine of the number passed in.</li> </ul></div> <div class="section"> -<h3><a name="SQRT"></a><tt>SQRT</tt></h3> +<h3><a name="SPLIT"></a><tt>SPLIT</tt></h3> <ul> -<li>Description: Returns the square root of a number.</li> +<li>Description: Splits the string by the delimiter.</li> <li>Input: <ul> -<li>number - The number to take the square root of</li> +<li>input - String to split</li> + +<li>delim - String delimiter</li> </ul></li> -<li>Returns: The square root of the number passed in.</li> +<li>Returns: List of strings</li> </ul></div> <div class="section"> -<h3><a name="STRING_ENTROPY"></a><tt>STRING_ENTROPY</tt></h3> +<h3><a name="SQRT"></a><tt>SQRT</tt></h3> <ul> -<li>Description: Computes the base-2 shannon entropy of a string.</li> +<li>Description: Returns the square root of a number.</li> <li>Input: <ul> -<li>input - String</li> +<li>number - The number to take the square root of</li> </ul></li> -<li>Returns: The base-2 shannon entropy of the string (<a class="externalLink" href="https://en.wikipedia.org/wiki/Entropy_(information_theory)#Definition)">https://en.wikipedia.org/wiki/Entropy_(information_theory)#Definition)</a>. The unit of this is bits.</li> +<li>Returns: The square root of the number passed in.</li> </ul></div> <div class="section"> -<h3><a name="SPLIT"></a><tt>SPLIT</tt></h3> +<h3><a name="STRING_ENTROPY"></a><tt>STRING_ENTROPY</tt></h3> <ul> -<li>Description: Splits the string by the delimiter.</li> +<li>Description: Computes the base-2 shannon entropy of a string.</li> <li>Input: <ul> -<li>input - String to split</li> - -<li>delim - String delimiter</li> +<li>input - String</li> </ul></li> -<li>Returns: List of strings</li> +<li>Returns: The base-2 shannon entropy of the string (<a class="externalLink" href="https://en.wikipedia.org/wiki/Entropy_(information_theory)#Definition)">https://en.wikipedia.org/wiki/Entropy_(information_theory)#Definition)</a>. The unit of this is bits.</li> </ul></div> <div class="section"> <h3><a name="STARTS_WITH"></a><tt>STARTS_WITH</tt></h3> @@ -2773,6 +2958,24 @@ <li>Returns: The tangent of the number passed in.</li> </ul></div> <div class="section"> +<h3><a name="TLSH_DIST"></a><tt>TLSH_DIST</tt></h3> + +<ul> + +<li>Description: Will return the hamming distance between two TLSH hashes (note: must be computed with the same params). For more information, see <a class="externalLink" href="https://github.com/trendmicro/tlsh";>https://github.com/trendmicro/tlsh</a> and Jonathan Oliver, Chun Cheng, and Yanggui Chen, TLSH - A Locality Sensitive Hash. 4th Cybercrime and Trustworthy Computing Workshop, Sydney, November 2013. For a discussion of tradeoffs, see Table II on page 5 of <a class="externalLink" href="https://github.com/trendmicro/tlsh/blob/master/TLSH_CTC_final.pdf";>https://github.com/trendmicro/tlsh/blob/master/TLSH_CTC_final.pdf</a></li> + +<li>Input: + +<ul> + +<li>hash1 - The first TLSH hash</li> + +<li>hash2 - The first TLSH hash</li> + +<li>includeLength? - Include the length in the distance calculation or not? Returns: An integer representing the distance between hash1 and hash2. The distance is roughly hamming distance, so 0 is very similar.</li> + </ul></li> +</ul></div> +<div class="section"> <h3><a name="TO_DOUBLE"></a><tt>TO_DOUBLE</tt></h3> <ul> @@ -2809,7 +3012,7 @@ <li>Returns: Epoch timestamp</li> </ul></div> <div class="section"> -<h3><a name="TO_FOAT"></a><tt>TO_FOAT</tt></h3> +<h3><a name="TO_FLOAT"></a><tt>TO_FLOAT</tt></h3> <ul> @@ -2841,6 +3044,54 @@ <li>Returns: Integer version of the first argument</li> </ul></div> <div class="section"> +<h3><a name="TO_JSON_LIST"></a><tt>TO_JSON_LIST</tt></h3> + +<ul> + +<li>Description: Accepts JSON string as an input and returns a List object parsed by Jackson. You need to be aware of content of JSON string that is to be parsed. For e.g. <tt>GET_FIRST( TO_JSON_LIST( '[ "foo", 2]')</tt> would yield <tt>foo</tt></li> + +<li>Input: + +<ul> + +<li>string - The JSON string to be parsed</li> + </ul></li> + +<li>Returns: A parsed List object</li> +</ul></div> +<div class="section"> +<h3><a name="TO_JSON_MAP"></a><tt>TO_JSON_MAP</tt></h3> + +<ul> + +<li>Description: Accepts JSON string as an input and returns a Map object parsed by Jackson. You need to be aware of content of JSON string that is to be parsed. For e.g. <tt>MAP_GET( 'bar', TO_JSON_MAP( '{ "foo" : 1, "bar" : 2}' )</tt> would yield <tt>2</tt></li> + +<li>Input: + +<ul> + +<li>string - The JSON string to be parsed</li> + </ul></li> + +<li>Returns: A parsed Map object</li> +</ul></div> +<div class="section"> +<h3><a name="TO_JSON_OBJECT"></a><tt>TO_JSON_OBJECT</tt></h3> + +<ul> + +<li>Description: Accepts JSON string as an input and returns a JSON Object parsed by Jackson. You need to be aware of content of JSON string that is to be parsed. For e.g. <tt>MAP_GET( 'bar', TO_JSON_OBJECT( '{ "foo" : 1, "bar" : 2}' )</tt> would yield <tt>2</tt></li> + +<li>Input: + +<ul> + +<li>string - The JSON string to be parsed</li> + </ul></li> + +<li>Returns: A parsed JSON object</li> +</ul></div> +<div class="section"> <h3><a name="TO_LONG"></a><tt>TO_LONG</tt></h3> <ul> @@ -3043,7 +3294,7 @@ <ul> -<li>list* - Lists to zip.</li> +<li>list(s) - List(s) to zip.</li> </ul></li> <li>Returns: The zip of the lists. The returned list is the min size of all the lists. e.g. <tt>ZIP( [ 1, 2 ], [ 3, 4, 5] ) == [ [1, 3], [2, 4] ]</tt></li> @@ -3059,7 +3310,7 @@ <ul> -<li>list* - Lists to zip.</li> +<li>list(s) - List(s) to zip.</li> </ul></li> <li>Returns: The zip of the lists. The returned list is the max size of all the lists. Empty elements are null e.g. <tt>ZIP_LONGEST( [ 1, 2 ], [ 3, 4, 5] ) == [ [1, 3], [2, 4], [null, 5] ]</tt></li> @@ -3171,6 +3422,16 @@ mvn exec:java -Dexec.mainClass="org.apache.metron.stellar.common.benchmark. </ul> <p>Note: Stellar classpath configuration from the global config is honored here if the REPL knows about zookeeper.</p> <div class="section"> +<h3><a name="Environment_Variables"></a>Environment Variables</h3> +<p>When starting the REPL via <tt>$METRON_HOME/bin/stellar</tt> you can specify certain environment variables to customize the experience:</p> + +<ul> + +<li><tt>JVMFLAGS</tt> - Arbitrary JVM flags to pass to the <tt>java</tt> command when starting the REPL.</li> + +<li><tt>CONTRIB</tt> - Directory where jars with Stellar functions can be placed. The default is <tt>$METRON_HOME/contrib</tt>.</li> +</ul></div> +<div class="section"> <h3><a name="Getting_Started"></a>Getting Started</h3> <p>To run the Stellar Shell from within a deployed Metron cluster, run the following command on the host where Metron is installed.</p> @@ -3270,15 +3531,44 @@ IN_SUBNET <div class="source"> <div class="source"> -<pre>Stellar, Go! -{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH} -[Stellar]>>> %vars +<pre>[Stellar]>>> %vars [Stellar]>>> foo := 2 + 2 4.0 [Stellar]>>> %vars foo = 4.0 </pre></div></div></div> <div class="section"> +<h4><a name="aglobals"></a><tt>%globals</tt></h4> +<p>Lists all values that are defined in the global configuration.</p> +<p>Most of Metron’s functional components have access to what is called the global configuration. This is a key/value configuration store that can be used to customize Metron. Many Stellar functions accept configuration values from the global configuration. The Stellar Shell also leverages the global configuration for customizing the behavior of many Stellar functions. </p> + +<div class="source"> +<div class="source"> +<pre>[Stellar]>>> %globals +{es.clustername=metron, es.ip=node1:9300, es.date.format=yyyy.MM.dd.HH, parser.error.topic=indexing, update.hbase.table=metron_update, update.hbase.cf=t} +</pre></div></div></div> +<div class="section"> +<h4><a name="adefine"></a><tt>%define</tt></h4> +<p>Defines a global configuration value in the current shell session. This value will be forgotten once the session is ended.</p> + +<div class="source"> +<div class="source"> +<pre>[Stellar]>>> %define bootstrap.servers := "node1:6667" +node1:6667 +[Stellar]>>> %globals +{bootstrap.servers=node1:6667} +</pre></div></div></div> +<div class="section"> +<h4><a name="aundefine"></a><tt>%undefine</tt></h4> +<p>Undefine a global configuration value in the current shell session. This will not modify the persisted global configuration.</p> + +<div class="source"> +<div class="source"> +<pre>[Stellar]>>> %undefine bootstrap.servers +[Stellar]>>> %globals +{} +</pre></div></div></div> +<div class="section"> <h4><a name="afunction"></a><tt>?<function></tt></h4> <p>Returns formatted documentation of the Stellar function. Provides the description of the function along with the expected arguments.</p> @@ -3326,9 +3616,9 @@ Please note that functions are loading lazily in the background and will be unav [Stellar]>>> Functions loaded, you may refer to functions now... %functions ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYS TEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR -</pre></div></div> -<p><a name="Stellar_Configuration"></a></p> -<h1>Stellar Configuration</h1> +</pre></div></div></div></div> +<div class="section"> +<h2><a name="Stellar_Configuration"></a>Stellar Configuration</h2> <p>Stellar can be configured in a variety of ways from the <a href="../../metron-platform/metron-common/index.html#Global_Configuration">Global Configuration</a>. In particular, there are three main configuration parameters around configuring Stellar:</p> <ul> @@ -3338,7 +3628,7 @@ ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, C <li><tt>stellar.function.resolver.includes</tt></li> <li><tt>stellar.function.resolver.excludes</tt></li> -</ul></div></div> +</ul></div> <div class="section"> <h2><a name="stellar.function.paths"></a><tt>stellar.function.paths</tt></h2> <p>If specified, Stellar will use a custom classloader which will wrap the context classloader and allow for the resolution of classes stored in jars not shipped with Metron and stored in a variety of mediums:</p> @@ -3366,7 +3656,7 @@ ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, C <div class="source"> <pre>{ ... - "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/metron-management-0.4.1.jar, hdfs://node1:8020/apps/metron/3rdparty/.*.jar" + "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/metron-management-0.4.2.jar, hdfs://node1:8020/apps/metron/3rdparty/.*.jar" } </pre></div></div> <p>Please be aware that this classloader does not reload functions dynamically and the classpath specified here in the global config is read on topology start. A change in classpath, to be picked up, would necessitate a topology restart at the moment</p></div> @@ -3396,7 +3686,7 @@ ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, C <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved.
