http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-analytics/metron-profiler-client/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-analytics/metron-profiler-client/index.html b/current-book/metron-analytics/metron-profiler-client/index.html index 54dd5d6..47cef0a 100644 --- a/current-book/metron-analytics/metron-profiler-client/index.html +++ b/current-book/metron-analytics/metron-profiler-client/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-02-23 + | Generated by Apache Maven Doxia at 2017-06-27 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Metron Profiler Client</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -30,14 +30,11 @@ <div class="container-fluid"> <div id="banner"> <div class="pull-left"> - <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> </a> </div> - <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> - <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> - </a> - </div> + <div class="pull-right"> </div> <div class="clear"><hr/></div> </div> @@ -51,8 +48,8 @@ </li> <li class="divider ">/</li> <li class=""> - <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> - Metron-Incubating</a> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> </li> <li class="divider ">/</li> <li class=""> @@ -64,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> </ul> </div> @@ -78,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -127,7 +124,7 @@ </li> </ul> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -141,14 +138,28 @@ <i class="none"></i> Docker</a> </li> - + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + <li> <a href="../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> @@ -182,25 +193,9 @@ <p>This project provides a client API for accessing the profiles generated by the <a href="../metron-profiler/index.html">Metron Profiler</a>. This includes both a Java API and Stellar API for accessing the profile data. The primary use case is to extract profile data for use during model scoring.</p> <div class="section"> <h2><a name="Stellar_Client_API"></a>Stellar Client API</h2> -<p>The following are usage examples that show how the Stellar API can be used to read profiles generated by the <a href="../metron-profiler/index.html">Metron Profiler</a>. This API would be used in conjunction with other Stellar functions like <a href="../../metron-platform/metron-common/index.html#MAAS_MODEL_APPLY"><tt>MAAS_MODEL_APPLY</tt></a> to perform model scoring on streaming data.</p> -<p>These examples assume a profile has been defined called ‘snort-alerts’ that tracks the number of Snort alerts associated with an IP address over time. The profile definition might look similar to the following.</p> - -<div class="source"> -<div class="source"> -<pre>{ - "profiles": [ - { - "profile": "snort-alerts", - "foreach": "ip_src_addr", - "onlyif": "source.type == 'snort'", - "update": { "s": "STATS_ADD(s, 1)" }, - "result": "STATS_MEAN(s)" - } - ] -} -</pre></div></div> -<p>During model scoring the entity being scored, in this case a particular IP address, will be known. The following examples shows how this profile data might be retrieved.</p> -<p>The Stellar client consists of the <tt>PROFILE_GET</tt> command, which takes the following arguments:</p> +<div class="section"> +<h3><a name="PROFILE_GET"></a><tt>PROFILE_GET</tt></h3> +<p>The <tt>PROFILE_GET</tt> command allows you to select all of the profile measurements written. This command takes the following arguments:</p> <div class="source"> <div class="source"> @@ -217,21 +212,8 @@ OPTIONAL: </pre></div></div> <p>There is an older calling format where <tt>groups_list</tt> is specified as a sequence of group names, “varargs” style, instead of a List object. This format is still supported for backward compatibility, but it is deprecated, and it is disallowed if the optional <tt>config_overrides</tt> argument is used.</p> <p>The <tt>periods</tt> field is (likely) the output of another Stellar function which defines the times to include.</p> -<p><tt>PROFILE_FIXED</tt>: The profiler periods associated with a fixed lookback starting from now. These are ProfilePeriod objects.</p> - -<div class="source"> -<div class="source"> -<pre>REQUIRED: - durationAgo - How long ago should values be retrieved from? - units - The units of 'durationAgo'. -OPTIONAL: - config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter - of the same name. Default is the empty Map, meaning no overrides. - -e.g. To retrieve all the profiles for the last 5 hours. PROFILE_GET('profile', 'entity', PROFILE_FIXED(5, 'HOURS')) -</pre></div></div> <div class="section"> -<h3><a name="Groups_list_argument"></a>Groups_list argument</h3> +<h4><a name="Groups_list_argument"></a>Groups_list argument</h4> <p>The <tt>groups_list</tt> argument in the client must exactly correspond to the <a href="../metron-profiler/index.html#groupBy"><tt>groupBy</tt></a> configuration in the profile definition. If <tt>groupBy</tt> was not used in the profile, <tt>groups_list</tt> must be empty in the client. If <tt>groupBy</tt> was used in the profile, then the client <tt>groups_list</tt> is <b>not</b> optional; it must be the same length as the <tt>groupBy</tt> list, and specify exactly one selected group value for each <tt>groupBy</tt> criterion, in the same order. For example:</p> <div class="source"> @@ -241,7 +223,7 @@ Then in PROFILE_GET, an allowed groups value would be: [ “3”, &#x which will select only records from Tuesdays with port number 8080. </pre></div></div></div> <div class="section"> -<h3><a name="Configuration_and_the_config_overrides_argument"></a>Configuration and the config_overrides argument</h3> +<h4><a name="Configuration_and_the_config_overrides_argument"></a>Configuration and the config_overrides argument</h4> <p>By default, the Profiler creates profiles with a period duration of 15 minutes. This means that data is accumulated, summarized and flushed every 15 minutes. The Client API must also have knowledge of this duration to correctly retrieve the profile data. If the Client is expecting 15 minute periods, it will not be able to read data generated by a Profiler that was configured for 1 hour periods, and will return zero results. </p> <p>Similarly, all six Client configuration parameters listed in the table below must match the Profiler configuration parameter settings from the time the profile was created. The period duration and other configuration parameters from the Profiler topology are stored in local filesystem at <tt>$METRON_HOME/config/profiler.properties</tt>. The Stellar Client API can be configured correspondingly by setting the following properties in Metron’s global configuration, on local filesystem at <tt>$METRON_HOME/config/zookeeper/global.json</tt>, then uploaded to Zookeeper (at <tt>/metron/topology/global</tt>) by using <tt>zk_load_configs.sh</tt>: </p> @@ -334,43 +316,433 @@ which will select only records from Tuesdays with port number 8080. <td> </td> </tr> </tbody> -</table></div> +</table></div></div> +<div class="section"> +<h3><a name="Profile_Selectors"></a>Profile Selectors</h3> +<p>You will notice that the third argument for <tt>PROFILE_GET</tt> is a list of <tt>ProfilePeriod</tt> objects. This list is expected to be produced by another Stellar function. There are a couple options available.</p> +<div class="section"> +<h4><a name="PROFILE_FIXED"></a><tt>PROFILE_FIXED</tt></h4> +<p>The profiler periods associated with a fixed lookback starting from now. These are ProfilePeriod objects.</p> + +<div class="source"> +<div class="source"> +<pre>REQUIRED: + durationAgo - How long ago should values be retrieved from? + units - The units of 'durationAgo'. +OPTIONAL: + config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter + of the same name. Default is the empty Map, meaning no overrides. + +e.g. To retrieve all the profiles for the last 5 hours. PROFILE_GET('profile', 'entity', PROFILE_FIXED(5, 'HOURS')) +</pre></div></div> +<p>Note that the <tt>config_overrides</tt> parameter operates exactly as the <tt>config_overrides</tt> argument in <tt>PROFILE_GET</tt>. The only available parameters for override are:</p> + +<ul> + +<li><tt>profiler.client.period.duration</tt></li> + +<li><tt>profiler.client.period.duration.units</tt></li> +</ul></div> +<div class="section"> +<h4><a name="PROFILE_WINDOW"></a><tt>PROFILE_WINDOW</tt></h4> +<p><tt>PROFILE_WINDOW</tt> is intended to provide a finer-level of control over selecting windows for profiles:</p> + +<ul> + +<li>Specify windows relative to the data timestamp (see the optional <tt>now</tt> parameter below)</li> + +<li>Specify non-contiguous windows to better handle seasonal data (e.g. the last hour for every day for the last month)</li> + +<li>Specify profile output excluding holidays</li> + +<li>Specify only profile output on a specific day of the week</li> +</ul> +<p>It does this by a domain specific language mimicking natural language that defines the windows excluded.</p> + +<div class="source"> +<div class="source"> +<pre>REQUIRED: + windowSelector - The statement specifying the window to select. + now - Optional - The timestamp to use for now. +OPTIONAL: + config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter + of the same name. Default is the empty Map, meaning no overrides. + +e.g. To retrieve all the measurements written for 'profile' and 'entity' for the last hour +on the same weekday excluding weekends and US holidays across the last 14 days: +PROFILE_GET('profile', 'entity', PROFILE_WINDOW('1 hour window every 24 hours starting from 14 days ago including the current day of the week excluding weekends, holidays:us')) +</pre></div></div> +<p>Note that the <tt>config_overrides</tt> parameter operates exactly as the <tt>config_overrides</tt> argument in <tt>PROFILE_GET</tt>. The only available parameters for override are:</p> + +<ul> + +<li><tt>profiler.client.period.duration</tt></li> + +<li><tt>profiler.client.period.duration.units</tt></li> +</ul> +<div class="section"> +<h5><a name="The_Profile_Selector_Language"></a>The Profile Selector Language</h5> +<p>The domain specific language can be broken into a series of clauses, some optional</p> + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:blue">Total Temporal Duration</span></a> - The total range of time in which windows may be specified</li> + +<li><a href="#InclusionExclusion_specifiers"><span style="color:red">Temporal Window Width</span></a> - How large each temporal window</li> + +<li><a href="#Skip_distance"><span style="color:green">Skip distance</span></a> (optional)- How far to skip between when one window starts and when the next begins</li> + +<li><a href="#InclusionExclusion_specifiers"><span style="color:purple">Inclusion/Exclusion specifiers</span></a> (optional) - The set of specifiers to further filter the window</li> +</ul> +<p>One <i>must</i> specify either a total temporal duration or a temporal window width. The remaining clauses are optional. During the course of the following discussion, we will color code the clauses in the examples and link them to the relevant section for more detail.</p> +<p>From a high level, the language fits the following three forms, which are composed of the clauses above:</p> + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:red">time_interval WINDOW?</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">(INCLUDING specifier_list)? (EXCLUDING specifier_list)?</span></a></li> + +<li><a href="#Temporal_Window_Width"><span style="color:red">time_interval WINDOW?</span></a> <a href="#Skip_distance"><span style="color:green">EVERY time_interval</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">FROM time_interval (TO time_interval)?</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">(INCLUDING specifier_list)? (EXCLUDING specifier_list)?</span></a></li> + +<li><a href="#Total_Temporal_Duration"><span style="color:blue">FROM time_interval (TO time_interval)?</span></a></li> +</ul> +<div class="section"> +<h6><a name="Total_Temporal_Duration"></a><span style="color:blue">Total Temporal Duration</span></h6> +<p>Total temporal duration is specified by a phrase: <tt>FROM time_interval AGO TO time_interval AGO</tt> This indicates the beginning and ending of a time interval. This is an inclusive duration.</p> + +<ul> + +<li><tt>FROM</tt> - Can be the words “from” or “starting from”</li> + +<li><tt>time_interval</tt> - A time amount followed by a unit (e.g. 1 hour). Fractional amounts are not supported. The unit may be “minute”, “day”, “hour” with any pluralization.</li> + +<li><tt>TO</tt> - Can be the words “until” or “to”</li> + +<li><tt>AGO</tt> - Optionally the word “ago”</li> +</ul> +<p>The <tt>TO time_interval AGO</tt> portion is optional. If unspecified then it is expected that the time interval ends now.</p> +<p>Due to the vagaries of the english language, the from and the to portions, if both specified, are interchangeable with regard to which one specifies the start and which specifies the end. </p> +<p>In other words “<a href="#Total_Temporal_Duration"><span style="color:blue">starting from 1 hour ago to 30 minutes ago</span></a>” and “<a href="#Total_Temporal_Duration"><span style="color:blue">starting from 30 minutes ago to 1 hour ago</span></a>” specify the same temporal duration.</p> +<p><b>Examples</b></p> + +<ul> + +<li>A duration starting 1 hour ago and ending now + +<ul> + +<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 1 hour ago</span></a></li> + +<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 1 hour</span></a></li> + +<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from 1 hour ago</span></a></li> + +<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from 1 hour</span></a></li> + </ul></li> + +<li>A duration starting 1 hour ago and ending 30 minutes ago: + +<ul> + +<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 1 hour ago until 30 minutes ago</span></a></li> + +<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 30 minutes ago until 1 hour ago</span></a></li> + +<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from 1 hour ago to 30 minutes ago</span></a></li> + +<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from 1 hour to 30 minutes</span></a></li> + </ul></li> +</ul></div> +<div class="section"> +<h6><a name="Temporal_Window_Width"></a><span style="color:red">Temporal Window Width</span></h6> +<p>Temporal window width is the specification of a window. A window is may either repeat within total temporal duration or may fill the total temporal duration. This is an inclusive window. A window is specified by the phrase: <tt>time_interval WINDOW</tt></p> + +<ul> + +<li><tt>time_interval</tt> - A time amount followed by a unit (e.g. 1 hour). Fractional amounts are not supported. The unit may be “minute”, “day”, “hour” with any pluralization.</li> + +<li><tt>WINDOW</tt> - Optionally the word “window”</li> +</ul> +<p><b>Examples</b></p> + +<ul> + +<li>A fixed window starting 2 hours ago and going until now + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:red">2 hour</span></a></li> + +<li><a href="#Temporal_Window_Width"><span style="color:red">2 hours</span></a></li> + +<li><a href="#Temporal_Window_Width"><span style="color:red">2 hours window</span></a></li> + </ul></li> + +<li>A repeating 30 minute window starting 2 hours ago and repeating every hour until now. This would result in 2 30-minute wide windows: 2 hours ago and 1 hour ago + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">starting from 2 hours ago</span></a></li> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute windows</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 2 hours ago</span></a></li> + </ul></li> + +<li>A repeating 30 minute window starting 2 hours ago and repeating every hour until 30 minutes ago. This would result in 2 30-minute wide windows: 2 hours ago and 1 hour ago + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">starting from 2 hours ago until 30 minutes ago</span></a></li> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 2 hours ago to 30 minutes ago</span></a></li> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes window</span></a> <a href="#Skip_distance"><span style="color:green">for every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 30 minutes ago to 2 hours ago</span></a></li> + </ul></li> +</ul></div> +<div class="section"> +<h6><a name="Skip_distance"></a><span style="color:green">Skip distance</span></h6> +<p>Skip distance is the amount of time between temporal window beginnings that the next window starts. It is, in effect, the window period. </p> +<p>It is specified by the phrase <tt>EVERY time_interval</tt></p> + +<ul> + +<li><tt>time_interval</tt> - A time amount followed by a unit (e.g. 1 hour). Fractional amounts are not supported. The unit may be “minute”, “day”, “hour” with any pluralization.</li> + +<li><tt>EVERY</tt> - The word/phrase “every” or “for every”</li> +</ul> +<p><b>Examples</b></p> + +<ul> + +<li>A repeating 30 minute window starting 2 hours ago and repeating every hour until now. This would result in 2 30-minute wide windows: 2 hours ago and 1 hour ago + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">starting from 2 hours ago </span></a></li> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 2 hours ago </span></a></li> + </ul></li> + +<li>A repeating 30 minute window starting 2 hours ago and repeating every hour until 30 minutes ago. This would result in 2 30-minute wide windows: 2 hours ago and 1 hour ago + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">starting from 2 hours ago until 30 minutes ago</span></a></li> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 2 hours ago to 30 minutes ago</span></a></li> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes window</span></a> <a href="#Skip_distance"><span style="color:green">for every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 30 minutes ago to 2 hours ago</span></a></li> + </ul></li> +</ul></div> +<div class="section"> +<h6><a name="InclusionExclusion_specifiers"></a><span style="color:purple">Inclusion/Exclusion specifiers</span></h6> +<p>Inclusion and Exclusion specifiers operate as filters on the set of windows. They operate on the window beginning timestamp.</p> +<p>For inclusion specifiers, windows who are passed by <i>any</i> of the set of inclusion specifiers are included.<br />inclusion specifiers. Similarly, windows who are passed by <i>any</i> of the set of exclusion specifiers are excluded. Exclusion specifiers trump inclusion specifiers.</p> +<p>Specifiers follow one of the following formats depending on if it is an inclusion or exclusion specifier:</p> + +<ul> + +<li><tt>INCLUSION specifier, specifier, ...</tt> + +<ul> + +<li><tt>INCLUSION</tt> can be “include”, “includes” or “including”</li> + </ul></li> + +<li><tt>EXCLUSION specifier, specifier, ...</tt> + +<ul> + +<li><tt>EXCLUSION</tt> can be “exclude”, “excludes” or “excluding”</li> + </ul></li> +</ul> +<p>The specifiers are a set of fixed specifiers available as part of the language:</p> + +<ul> + +<li>Fixed day of week-based specifiers - includes or excludes if the window is on the specified day of the week + +<ul> + +<li>“monday” or “mondays”</li> + +<li>“tuesday” or “tuesdays”</li> + +<li>“wednesday” or “wednesdays”</li> + +<li>“thursday” or “thursdays”</li> + +<li>“friday” or “fridays”</li> + +<li>“saturday” or “saturdays”</li> + +<li>“sunday” or “sundays”</li> + +<li>“weekday” or “weekdays”</li> + +<li>“weekend” or "“weekends”</li> + </ul></li> + +<li>Relative day of week-based specifiers - includes or excludes based on the day of week relative to now + +<ul> + +<li>“current day of the week”</li> + +<li>“current day of week”</li> + +<li>“this day of the week”</li> + +<li>“this day of week”</li> + </ul></li> + +<li>Specified date - includes or excludes based on the specified date + +<ul> + +<li>“date” - Takes up to 2 arguments + +<ul> + +<li>The day in <tt>yyyy/MM/dd</tt> format if no second argument is provided</li> + +<li>Optionally the format to specify the first argument in</li> + +<li>Example: <tt>date:2017/12/25</tt> would include or exclude December 25, 2017</li> + +<li>Example: <tt>date:20171225:yyyyMMdd</tt> would include or exclude December 25, 2017</li> + </ul></li> + </ul></li> + +<li>Holidays - includes or excludes based on if the window starts during a holiday + +<ul> + +<li>“holiday” or “holidays” + +<ul> + +<li>Arguments form the jollyday hierarchy of holidays. e.g. “us:nyc” would be holidays for New York City, USA</li> + +<li>If none is specified, it will choose based on locale.</li> + +<li>Countries supported are those supported in <a class="externalLink" href="https://github.com/svendiedrichsen/jollyday/tree/master/src/main/resources/holidays";>jollyday</a></li> + +<li>Example: <tt>holiday:us:nyc</tt> would be the holidays of New York City, USA</li> + +<li>Example: <tt>holiday:hu</tt> would be the holidays of Hungary</li> + </ul></li> + </ul></li> +</ul> +<p><b>WARNING: Daylight Savings Time effects</b></p> +<p>While Universal Time (UTC) is nice and constant, many servers are set to local timezones that enable Daylight Savings Time (DST). This means that twice a year, on DST transition weekends, “Sunday” is either 23 or 25 hours long. However, durations specified as “7 days ago” are always interpreted as “7*24 hours ago”. This can lead to some surprising effects when using days of the week as inclusion or exclusion specifiers.</p> +<p>For example, the profile window specified by the phrase “30 minute window every 24 hours from 7 days ago” will always have 7 thirty-minute intervals, and these will normally occur on 5 weekdays and 2 weekend days. However, if you invoke this window at 12:15am any day during the week following the start of DST, you will get these intervals (supposing you start early on a Wednesday morning):</p> + +<div class="source"> +<div class="source"> +<pre>Tuesday 12:15am-12:45am (yesterday) +Monday 12:15am-12:45am +Saturday 11:15pm-11:45pm (skipped Sunday!) +Friday 11:15pm-11:45pm +Thursday 11:15pm-11:45pm +Wednesday 11:15pm-11:45pm +Tuesday 11:15pm-11:45pm +</pre></div></div> +<p>Sunday got skipped over because it was only 23 hours long; that is, there were 24 hours between Saturday 11:15pm and Monday 12:15am. So if you specified “excluding weekends”, you would get 6 days’ intervals instead of the expected 5. There are multiple variations on this theme.</p> +<p>Remember that the underlying time is kept in UTC, so the data is always correct. It is only when attempting to interpret UTC as local time, date, and day, that these confusions may occur. They may be eliminated by setting your server timezone to UTC, or otherwise disabling DST.</p> +<p><b>Examples</b></p> +<p>Assume these are executed at noon.</p> + +<ul> + +<li>A 1 hour window for the past 8 ‘current day of the week’ + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:red">1 hour window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 56 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">including this day of the week</span></a></li> + </ul></li> + +<li>A 1 hour window for the past 8 tuesdays + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:red">1 hour window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 56 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">including tuesdays</span></a></li> + </ul></li> + +<li>A 30 minute window every tuesday at noon starting 14 days ago until now + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 14 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">including tuesdays</span></a></li> + </ul></li> + +<li>A 30 minute window every day except holidays and weekends at noon starting 14 days ago until now + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 14 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">excluding holidays:us, weekends</span></a></li> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 14 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">including weekdays excluding holidays:us, weekends</span></a></li> + </ul></li> + +<li>A 30 minute window at noon every day from 7 days ago including saturdays and excluding weekends. Because exclusions trump inclusions, the following will never yield any windows + +<ul> + +<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 7 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">including saturdays excluding weekends</span></a></li> + </ul></li> +</ul></div></div></div></div> <div class="section"> <h3><a name="Errors"></a>Errors</h3> -<p>The most common result of incorrect PROFILE_GET arguments or Client configuration parameters is an empty result set, rather than an error. The Client cannot effectively validate the arguments, because the Profiler configuration parameters may be changed and the profile itself does not store them. The person doing the querying must carry forward the knowledge of the Profiler configuration parameters from the time of profile creation, and use corresponding PROFILE_GET arguments and Client configuration parameters when querying the data.</p></div> +<p>The most common result of incorrect <tt>PROFILE_GET</tt> arguments or Client configuration parameters is an empty result set, rather than an error. The Client cannot effectively validate the arguments, because the Profiler configuration parameters may be changed and the profile itself does not store them. The person doing the querying must carry forward the knowledge of the Profiler configuration parameters from the time of profile creation, and use corresponding <tt>PROFILE_GET</tt> arguments and Client configuration parameters when querying the data.</p></div> <div class="section"> <h3><a name="Examples"></a>Examples</h3> -<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ over the past 4 hours.</p> +<p>The following are usage examples that show how the Stellar API can be used to read profiles generated by the <a href="../metron-profiler/index.html">Metron Profiler</a>. This API would be used in conjunction with other Stellar functions like <a href="../../metron-platform/metron-common/index.html#MAAS_MODEL_APPLY"><tt>MAAS_MODEL_APPLY</tt></a> to perform model scoring on streaming data.</p> +<p>These examples assume a profile has been defined called ‘snort-alerts’ that tracks the number of Snort alerts associated with an IP address over time. The profile definition might look similar to the following.</p> + +<div class="source"> +<div class="source"> +<pre>{ + "profiles": [ + { + "profile": "snort-alerts", + "foreach": "ip_src_addr", + "onlyif": "source.type == 'snort'", + "update": { "s": "STATS_ADD(s, 1)" }, + "result": "STATS_MEAN(s)" + } + ] +} +</pre></div></div> +<p>During model scoring the entity being scored, in this case a particular IP address, will be known. The following examples shows how this profile data might be retrieved. Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ over the past 4 hours.</p> <div class="source"> <div class="source"> -<pre>PROFILE_GET('snort-alerts', '10.0.0.1', 4, 'HOURS') +<pre>PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(4, 'HOURS')) </pre></div></div> <p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ over the past 2 days.</p> <div class="source"> <div class="source"> -<pre>PROFILE_GET('snort-alerts', '10.0.0.1', 2, 'DAYS') +<pre>PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(2, 'DAYS')) </pre></div></div> <p>If the profile had been defined to group the data by weekday versus weekend, then the following example would apply:</p> -<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ that occurred on ‘weekdays’ over the past month.</p> +<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ that occurred on ‘weekdays’ over the past 30 days.</p> <div class="source"> <div class="source"> -<pre>PROFILE_GET('snort-alerts', '10.0.0.1', 1, 'MONTHS', ['weekdays'] ) +<pre>PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(30, 'DAYS'), ['weekdays'] ) </pre></div></div> <p>The client may need to use a configuration different from the current Client configuration settings. For example, perhaps you are on a cluster shared with other analysts, and need to access a profile that was constructed 2 months ago using different period duration, while they are accessing more recent profiles constructed with the currently configured period duration. For this situation, you may use the <tt>config_overrides</tt> argument:</p> <p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ over the past 2 days, with no <tt>groupBy</tt>, and overriding the usual global client configuration parameters for window duration.</p> <div class="source"> <div class="source"> -<pre>PROFILE_GET('profile1', 'entity1', 2, 'DAYS', [], {'profiler.client.period.duration' : '2', 'profiler.client.period.duration.units' : 'MINUTES'}) +<pre>PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(2, 'DAYS', {'profiler.client.period.duration' : '2', 'profiler.client.period.duration.units' : 'MINUTES'}), []) </pre></div></div> -<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ that occurred on ‘weekdays’ over the past month, overriding the usual global client configuration parameters for window duration.</p> +<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ that occurred on ‘weekdays’ over the past 30 days, overriding the usual global client configuration parameters for window duration.</p> <div class="source"> <div class="source"> -<pre>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', ['weekdays'], {'profiler.client.period.duration' : '2', 'profiler.client.period.duration.units' : 'MINUTES'}) +<pre>PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(30, 'DAYS', {'profiler.client.period.duration' : '2', 'profiler.client.period.duration.units' : 'MINUTES'}), ['weekdays'] ) </pre></div></div></div></div> <div class="section"> <h2><a name="Getting_Started"></a>Getting Started</h2> @@ -414,8 +786,9 @@ Returns: The selected profile measurements. <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017. - All Rights Reserved. + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. </div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-analytics/metron-profiler/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-analytics/metron-profiler/index.html b/current-book/metron-analytics/metron-profiler/index.html index adb7ffb..2f727dc 100644 --- a/current-book/metron-analytics/metron-profiler/index.html +++ b/current-book/metron-analytics/metron-profiler/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-02-23 + | Generated by Apache Maven Doxia at 2017-06-27 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Metron Profiler</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -30,14 +30,11 @@ <div class="container-fluid"> <div id="banner"> <div class="pull-left"> - <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> </a> </div> - <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> - <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> - </a> - </div> + <div class="pull-right"> </div> <div class="clear"><hr/></div> </div> @@ -51,8 +48,8 @@ </li> <li class="divider ">/</li> <li class=""> - <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> - Metron-Incubating</a> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> </li> <li class="divider ">/</li> <li class=""> @@ -64,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> </ul> </div> @@ -78,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -127,7 +124,7 @@ </li> </ul> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -141,14 +138,28 @@ <i class="none"></i> Docker</a> </li> - + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + <li> <a href="../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> @@ -271,8 +282,13 @@ hbase(main):001:0> count 'profiler' <li> <p>Use the Profiler Client to read the profile data. The below example <tt>PROFILE_GET</tt> command will read data written by the sample profile given above, if 10.0.0.1 is one of the input values for <tt>ip_src_addr</tt>. More information on configuring and using the client can be found <a href="../metron-profiler-client/index.html">here</a>. It is assumed that the <tt>PROFILE_GET</tt> client is correctly configured before using it.</p> -<p>``` $ bin/stellar -z node1:2181</p> -<p>[Stellar]>>> PROFILE_GET( “test”, “10.0.0.1”, PROFILE_FIXED(30, “MINUTES”)) [451, 448] ```</p></li> + +<div class="source"> +<div class="source"> +<pre>$ bin/stellar -z node1:2181 +[Stellar]>>> PROFILE_GET( "test", "10.0.0.1", PROFILE_FIXED(30, "MINUTES")) +[451, 448] +</pre></div></div></li> </ol></div> <div class="section"> <h2><a name="Creating_Profiles"></a>Creating Profiles</h2> @@ -285,9 +301,9 @@ hbase(main):001:0> count 'profiler' <th>Name </th> -<th>Required </th> +<th> </th> -<th>Description </th> +<th>Description</th> </tr> </thead> <tbody> @@ -298,7 +314,7 @@ hbase(main):001:0> count 'profiler' <td>Required </td> -<td>Unique name identifying the profile. </td> +<td>Unique name identifying the profile.</td> </tr> <tr class="a"> @@ -307,7 +323,7 @@ hbase(main):001:0> count 'profiler' <td>Required </td> -<td>A separate profile is maintained “for each” of these. </td> +<td>A separate profile is maintained “for each” of these.</td> </tr> <tr class="b"> @@ -316,7 +332,7 @@ hbase(main):001:0> count 'profiler' <td>Optional </td> -<td>Boolean expression that determines if a message should be applied to the profile. </td> +<td>Boolean expression that determines if a message should be applied to the profile.</td> </tr> <tr class="a"> @@ -325,7 +341,7 @@ hbase(main):001:0> count 'profiler' <td>Optional </td> -<td>One or more Stellar expressions used to group the profile measurements when persisted. </td> +<td>One or more Stellar expressions used to group the profile measurements when persisted.</td> </tr> <tr class="b"> @@ -334,7 +350,7 @@ hbase(main):001:0> count 'profiler' <td>Optional </td> -<td>One or more expressions executed at the start of a window period. </td> +<td>One or more expressions executed at the start of a window period.</td> </tr> <tr class="a"> @@ -343,7 +359,7 @@ hbase(main):001:0> count 'profiler' <td>Required </td> -<td>One or more expressions executed when a message is applied to the profile. </td> +<td>One or more expressions executed when a message is applied to the profile.</td> </tr> <tr class="b"> @@ -352,7 +368,7 @@ hbase(main):001:0> count 'profiler' <td>Required </td> -<td>A Stellar expression that is executed when the window period expires. </td> +<td>Stellar expressions that are executed when the window period expires.</td> </tr> <tr class="a"> @@ -361,7 +377,7 @@ hbase(main):001:0> count 'profiler' <td>Optional </td> -<td>Profile data is purged after this period of time, specified in milliseconds. </td> +<td>Profile data is purged after this period of time, specified in milliseconds.</td> </tr> </tbody> </table> @@ -415,7 +431,49 @@ hbase(main):001:0> count 'profiler' <div class="section"> <h3><a name="result"></a><tt>result</tt></h3> <p><i>Required</i></p> -<p>A Stellar expression that is executed when the window period expires. The expression is expected to summarize the messages that were applied to the profile over the window period, using the state accumulated by the updates. The result will typically be a single numeric value, but it may be any serializable object, as shown in Example 4 below. </p></div> +<p>Stellar expressions that are executed when the window period expires. The expressions are expected to summarize the messages that were applied to the profile over the window period. In the most basic form a single result is persisted for later retrieval.</p> + +<div class="source"> +<div class="source"> +<pre>"result": "var1 + var2" +</pre></div></div> +<p>For more advanced use cases, a profile can generate two types of results. A profile can define one or both of these result types at the same time. </p> + +<ul> + +<li><tt>profile</tt>: A required expression that defines a value that is persisted for later retrieval.</li> + +<li><tt>triage</tt>: An optional expression that defines values that are accessible within the Threat Triage process.</li> +</ul> +<p><b>profile</b></p> +<p>A required Stellar expression that results in a value that is persisted in the profile store for later retrieval. The expression can result in any object that is Kryo serializable. These values can be retrieved for later use with the <a href="../metron-profiler-client/index.html">Profiler Client</a>. </p> + +<div class="source"> +<div class="source"> +<pre>"result": { + "profile": "2 + 2" +} +</pre></div></div> +<p>An alternative, simplified form is also acceptable.</p> + +<div class="source"> +<div class="source"> +<pre>"result": "2 + 2" +</pre></div></div> +<p><b>triage</b></p> +<p>An optional map of one or more Stellar expressions. The value of each expression is made available to the Threat Triage process under the given name. Each expression must result in a either a primitive type, like an integer, long, or short, or a String. All other types will result in an error.</p> +<p>In the following example, three values, the minimum, the maximum and the mean are appended to a message. This message is consumed by Metron, like other sources of telemetry, and each of these values are accessible from within the Threat Triage process using the given field names; <tt>min</tt>, <tt>max</tt>, and <tt>mean</tt>.</p> + +<div class="source"> +<div class="source"> +<pre>"result": { + "triage": { + "min": "STATS_MIN(stats)", + "max": "STATS_MAX(stats)", + "mean": "STATS_MEAN(stats)" + } +} +</pre></div></div></div> <div class="section"> <h3><a name="expires"></a><tt>expires</tt></h3> <p><i>Optional</i></p> @@ -431,7 +489,7 @@ hbase(main):001:0> count 'profiler' <th>Setting </th> -<th>Description </th> +<th>Description</th> </tr> </thead> <tbody> @@ -440,84 +498,91 @@ hbase(main):001:0> count 'profiler' <td>profiler.workers </td> -<td>The number of worker processes to create for the topology. </td> +<td>The number of worker processes to create for the topology.</td> </tr> <tr class="a"> <td>profiler.executors </td> -<td>The number of executors to spawn per component. </td> +<td>The number of executors to spawn per component.</td> </tr> <tr class="b"> <td>profiler.input.topic </td> -<td>The name of the Kafka topic from which to consume data. </td> +<td>The name of the Kafka topic from which to consume data.</td> </tr> <tr class="a"> -<td>profiler.period.duration </td> +<td>profiler.output.topic </td> -<td>The duration of each profile period. This value should be defined along with <tt>profiler.period.duration.units</tt>. </td> +<td>The name of the Kafka topic to which profile data is written. Only used with profiles that use the <a href="#result"><tt>triage</tt> result field</a>.</td> </tr> <tr class="b"> -<td>profiler.period.duration.units </td> +<td>profiler.period.duration </td> -<td>The units used to specify the <tt>profiler.period.duration</tt>. </td> +<td>The duration of each profile period. This value should be defined along with <tt>profiler.period.duration.units</tt>.</td> </tr> <tr class="a"> -<td>profiler.ttl </td> +<td>profiler.period.duration.units </td> -<td>If a message has not been applied to a Profile in this period of time, the Profile will be forgotten and its resources will be cleaned up. This value should be defined along with <tt>profiler.ttl.units</tt>. </td> +<td>The units used to specify the <tt>profiler.period.duration</tt>.</td> </tr> <tr class="b"> -<td>profiler.ttl.units </td> +<td>profiler.ttl </td> -<td>The units used to specify the <tt>profiler.ttl</tt>. </td> +<td>If a message has not been applied to a Profile in this period of time, the Profile will be forgotten and its resources will be cleaned up. This value should be defined along with <tt>profiler.ttl.units</tt>.</td> </tr> <tr class="a"> -<td>profiler.hbase.salt.divisor </td> +<td>profiler.ttl.units </td> -<td>A salt is prepended to the row key to help prevent hotspotting. This constant is used to generate the salt. Ideally, this constant should be roughly equal to the number of nodes in the Hbase cluster. </td> +<td>The units used to specify the <tt>profiler.ttl</tt>.</td> </tr> <tr class="b"> -<td>profiler.hbase.table </td> +<td>profiler.hbase.salt.divisor </td> -<td>The name of the HBase table that profiles are written to. </td> +<td>A salt is prepended to the row key to help prevent hotspotting. This constant is used to generate the salt. Ideally, this constant should be roughly equal to the number of nodes in the Hbase cluster.</td> </tr> <tr class="a"> -<td>profiler.hbase.column.family </td> +<td>profiler.hbase.table </td> -<td>The column family used to store profiles. </td> +<td>The name of the HBase table that profiles are written to.</td> </tr> <tr class="b"> -<td>profiler.hbase.batch </td> +<td>profiler.hbase.column.family </td> -<td>The number of puts that are written in a single batch. </td> +<td>The column family used to store profiles.</td> </tr> <tr class="a"> +<td>profiler.hbase.batch </td> + +<td>The number of puts that are written in a single batch.</td> + </tr> + +<tr class="b"> + <td>profiler.hbase.flush.interval.seconds </td> -<td>The maximum number of seconds between batch writes to HBase. </td> +<td>The maximum number of seconds between batch writes to HBase.</td> </tr> </tbody> </table> @@ -766,8 +831,9 @@ hbase(main):001:0> count 'profiler' <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017. - All Rights Reserved. + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. </div> http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-analytics/metron-statistics/HLLP.html ---------------------------------------------------------------------- diff --git a/current-book/metron-analytics/metron-statistics/HLLP.html b/current-book/metron-analytics/metron-statistics/HLLP.html index 7d389ab..0017621 100644 --- a/current-book/metron-analytics/metron-statistics/HLLP.html +++ b/current-book/metron-analytics/metron-statistics/HLLP.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-02-23 + | Generated by Apache Maven Doxia at 2017-06-27 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – HyperLogLogPlus</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -30,14 +30,11 @@ <div class="container-fluid"> <div id="banner"> <div class="pull-left"> - <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> </a> </div> - <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> - <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> - </a> - </div> + <div class="pull-right"> </div> <div class="clear"><hr/></div> </div> @@ -51,8 +48,8 @@ </li> <li class="divider ">/</li> <li class=""> - <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> - Metron-Incubating</a> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> </li> <li class="divider ">/</li> <li class=""> @@ -64,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> </ul> </div> @@ -78,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -136,7 +133,7 @@ </li> </ul> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -150,14 +147,28 @@ <i class="none"></i> Docker</a> </li> - + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + <li> <a href="../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> @@ -201,7 +212,7 @@ <p>Below is a table detailing the performance characteristics when running HLLP over multiple data set sizes. There are three primary variables manipulated during the test: cardinality, sparse set precision, and normal set precision. Each of these values will have an impact on error rates, execution time, and total memory consumption.</p></div> <div class="section"> <h3><a name="Choosing_p_sp"></a>Choosing p, sp</h3> -<p>There is a tradeoff when choosing values for p and sp. A higher value for sp means a lower error rate for instances when the sparse set is being used. However, a higher value for sp also requires more memory, which means the algorithm will cut over to the dense representation more quickly. The Google paper (listed above) recommends settings of p=25, sp=14 for optimal memory use and algorithm accuracy, and these are the defaults provided by the Apache Metron (Incubating) implementation in Stellar. You’ll notice in the tests below, memory consumption tops out around 12KB for this setting, regardless of cardinality size.</p></div> +<p>There is a tradeoff when choosing values for p and sp. A higher value for sp means a lower error rate for instances when the sparse set is being used. However, a higher value for sp also requires more memory, which means the algorithm will cut over to the dense representation more quickly. The Google paper (listed above) recommends settings of p=25, sp=14 for optimal memory use and algorithm accuracy, and these are the defaults provided by the Apache Metron implementation in Stellar. You’ll notice in the tests below, memory consumption tops out around 12KB for this setting, regardless of cardinality size.</p></div> <div class="section"> <h3><a name="Key"></a>Key</h3> <hr /> @@ -7574,8 +7585,9 @@ format err as %: true <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017. - All Rights Reserved. + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. </div> http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-analytics/metron-statistics/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-analytics/metron-statistics/index.html b/current-book/metron-analytics/metron-statistics/index.html index c167eaf..c663271 100644 --- a/current-book/metron-analytics/metron-statistics/index.html +++ b/current-book/metron-analytics/metron-statistics/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-02-23 + | Generated by Apache Maven Doxia at 2017-06-27 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Statistics and Mathematical Functions</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -30,14 +30,11 @@ <div class="container-fluid"> <div id="banner"> <div class="pull-left"> - <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> </a> </div> - <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> - <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> - </a> - </div> + <div class="pull-right"> </div> <div class="clear"><hr/></div> </div> @@ -51,8 +48,8 @@ </li> <li class="divider ">/</li> <li class=""> - <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> - Metron-Incubating</a> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> </li> <li class="divider ">/</li> <li class=""> @@ -64,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> </ul> </div> @@ -78,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -136,7 +133,7 @@ </li> </ul> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -150,14 +147,28 @@ <i class="none"></i> Docker</a> </li> - + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + <li> <a href="../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> @@ -225,7 +236,7 @@ <li>hyperLogLogPlus - the hllp set</li> </ul></li> -<li>Returns: Long value representing the cardinality for this set</li> +<li>Returns: Long value representing the cardinality for this set. Cardinality of a null set is 0.</li> </ul></div> <div class="section"> <h4><a name="HLLP_INIT"></a><tt>HLLP_INIT</tt></h4> @@ -259,7 +270,7 @@ <li>hllp - List of hllp estimators to merge. Takes a single hllp set or a list.</li> </ul></li> -<li>Returns: A new merged HyperLogLogPlus estimator set</li> +<li>Returns: A new merged HyperLogLogPlus estimator set. Passing an empty list returns null.</li> </ul></div></div> <div class="section"> <h3><a name="Mathematical_Functions"></a>Mathematical Functions</h3> @@ -800,8 +811,6 @@ if __name__ == '__main__': <div class="source"> <div class="source"> <pre>{ - "index": "mad", - "batchSize": 1, "enrichment": { "fieldMap": { "stellar" : { @@ -828,6 +837,23 @@ PROFILE_GET( 'sketchy_mad', 'global', PROFILE_FIXED(10, 'MINUTES')) ), value)&qu } } } +</pre></div></div> +<p>We also need an indexing configuration. Create the following in <tt>$METRON_HOME/config/zookeeper/indexing/mad.json</tt>:</p> + +<div class="source"> +<div class="source"> +<pre>{ + "hdfs" : { + "index": "mad", + "batchSize": 1, + "enabled" : true + }, + "elasticsearch" : { + "index": "mad", + "batchSize": 1, + "enabled" : true + } +} </pre></div></div></div> <div class="section"> <h4><a name="The_Profiler"></a>The Profiler</h4> @@ -903,8 +929,9 @@ PROFILE_GET( 'sketchy_mad', 'global', PROFILE_FIXED(10, 'MINUTES')) ), value)&qu <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017. - All Rights Reserved. + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. </div>
