This is an automated email from the ASF dual-hosted git repository.
poorejc pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-flagon.git
The following commit(s) were added to refs/heads/master by this push:
new b4dc9cc [FLAGON-415] Fixes broken scaling page
b4dc9cc is described below
commit b4dc9cc2c47c06359b400635d00113f31d8eeefd
Author: poorejc <[email protected]>
AuthorDate: Mon Jul 8 23:59:41 2019 -0400
[FLAGON-415] Fixes broken scaling page
---
content/community/index.html | 2 +-
content/docs/stack/scaling/index.html | 368 ++++++++++++++----------
content/docs/useralejs/dataschema/index.html | 2 +-
content/feed.xml | 2 +-
content/releases/index.html | 18 +-
site/_docs/stack/scaling.md | 174 ++++++++---
site/_site/community/index.html | 2 +-
site/_site/docs/stack/scaling/index.html | 368 ++++++++++++++----------
site/_site/docs/useralejs/dataschema/index.html | 2 +-
site/_site/feed.xml | 2 +-
site/_site/releases/index.html | 18 +-
11 files changed, 601 insertions(+), 357 deletions(-)
diff --git a/content/community/index.html b/content/community/index.html
index 6f7099a..75429cf 100644
--- a/content/community/index.html
+++ b/content/community/index.html
@@ -81,7 +81,7 @@
</h3>
<p>
For users of Apache Flagon who want to keep up to date with new releases
and community highlights. To subscribe, email <a
href="mailto:[email protected]";>[email protected]</a>.
- <a class="ui yellow button"
href="mailto:[email protected]";>User List</a>
+ <a class="ui yellow button"
href="mailto:[email protected]";>User List</a>
</p>
<h3 class="ui header">
diff --git a/content/docs/stack/scaling/index.html
b/content/docs/stack/scaling/index.html
index 6b544ab..da8e903 100644
--- a/content/docs/stack/scaling/index.html
+++ b/content/docs/stack/scaling/index.html
@@ -248,80 +248,132 @@
<h2 class="ui header">Scaling Considerations</h2>
<h3
id="scaling-apache-flagon-an-introduction-and-first-principles">Scaling Apache
Flagon: An Introduction and First Principles</h3>
-<p>This guide touches on some basic principles to keep in mind as you’re
planning for scale with <a
href="https://github.com/apache/incubator-flagon";>Apache Flagon</a>. We provide
some high-level guidance and considerations for working with an Elastic stack
to scale Flagon that are unique to <a href="/docs/useralejs">Apache
UserALE.js</a> data. We also walk through some of our benchmarking tools and
methodologies, so that you walk into discussions about scale with some accurate
assumptio [...]
+<p>This guide touches on some basic principles to keep in mind as you’re
planning for scale with <a
href="https://github.com/apache/incubator-flagon";>Apache Flagon</a>.</p>
+
+<p>We provide high-level guidance and considerations for working with an
Elastic stack to scale <a href="/docs/useralejs">Apache UserALE.js</a>
services.</p>
+
+<p>This guide also provides an overview of benchmarking tools and
methodologies for accurately gauging your scaling needs.</p>
<p><strong>“It Depends…“</strong></p>
-<p>The best way to scale Apache Flagon depends entirely on your use-case: how
you’ll use your Apache UserALE.js data and which <a
href="/docs/useralejs/dataschema">UserALE.js data streams</a> you’ll use. This
page provides a few Apache Flagon-specific considerations to think about as you
decide how best to scale, and some useful methods for benchmarking to help you
make that determination.</p>
+<p>The best way to scale Apache Flagon depends entirely on your use-case:</p>
+
+<ul>
+ <li>how you’ll use your Apache UserALE.js data;</li>
+ <li>which <a href="/docs/useralejs/dataschema">data streams</a> you’ll
use;</li>
+ <li>how long you need to keep your data.</li>
+</ul>
<p><strong>The Apache Flagon Single-Node Elastic Container is an Ingredient,
Not a Whole Solution</strong></p>
-<p>The single-node Elastic (ELK) stack in the Docker container distributed by
<a href="/docs/stack">Apache Flagon</a> is not alone suitable for most
production-level use-cases. It may be suitable on its own, for “grab-and-go”
user-testing use cases, for example. This would entail just a few days of data
collection from a specific application, from just a few users. But, alone it
will fail quickly for most persistent data collection uses and enterprise-scale
use-cases. Rather, this contai [...]
+<p>The single-node Elastic (ELK) distributed by <a href="/docs/stack">Apache
Flagon</a> is not alone suitable for most production-level use-cases.</p>
+
+<p>It may be suitable on its own for “grab-and-go” user-testing use cases;
just a few days of data collection from a specific application, from just a few
users.</p>
+
+<p>Alone it will fail quickly for any enterprise-scale use-cases.</p>
+
+<p>Instead, this container is meant to be a building block for larger
solutions:</p>
<ol>
- <li>Our ELK .yml config files for our Docker container can be used as the
building-blocks for your very own <a
href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c";>multi-node
cluster</a> with load-balancing capabilities.</li>
+ <li>Our ELK .yml config files for our Docker container can be used as the
building-blocks for your very own load-balanced, <a
href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c";>multi-node
cluster</a>.</li>
<li>You can use our <a
href="https://github.com/apache/incubator-flagon/tree/master/kubernetes";>Kubernetes
build</a>, which relies on our Docker assets, to scale your Apache Flagon
stack to meet your needs.</li>
<li>You can use our single-node container to scale out in <a
href="https://aws.amazon.com/elasticbeanstalk/";>AWS Elastic Beanstalk
(EBS)</a>.</li>
</ol>
<p><strong>Apache Flagon Data Also Scales</strong></p>
-<p>It’s important to note that the burden of scale isn’t placed wholly on your
Apache Flagon stack. Flagon’s behavioral logging capability, <a
href="/docs/useralejs">Apache UserALE.js</a> also scales. This means that one
of the most cost-efficient ways to manage the resources behind your Apache
Flagon stack, is to <a href="/docs/useralejs/API">configure</a> or <a
href="/docs/useralejs/modifying">modify</a> UserALE.js to meet your
use-case.</p>
+<p>Flagon’s behavioral logging capability, <a href="/docs/useralejs">Apache
UserALE.js</a> also scales. The most efficient way to manage scale and
resources, is to <a href="/docs/useralejs/API">configure</a> or <a
href="/docs/useralejs/modifying">modify</a> UserALE.js.</p>
<h3 id="sizing-up-an-elastic-stack">Sizing Up an Elastic Stack</h3>
-<p>When thinking about how to scale your Apache Flagon Elastic stack, it’s
important to note that Elasticsearch isn’t a database, its a datastore, which
stores documents. Elastic is built on top of <a
href="http://lucene.apache.org/";>Lucene</a>. That means that Apache Flagon
“logs” aren’t logs once they’re indexed in Elastic, they become searchable
documents. That’s a huge strength (and why we chose Elastic), but it also means
that assumptions about resource consumptions based purely on [...]
-
-<ol>
- <li>
- <p>Given that documents are the atomic unit of storage in Elastic,
<em>document generation rate</em> is the most important consideration to
scaling. Default <a href="/docs/useralejs">Apache UserALE.js parameters</a>
produce a lot of <a href="/docs/useralejs/dataschema">data</a>, even from
single users in Apache UserALE.js. In fact, we used say that “drinking from the
fire-hose” didn’t quite do our data-rate justice–we used to say that opening up
UserALE.js was more like “drinking fro [...]
- </li>
- <li>Your Elastic resource needs will also grow with <em>document
length</em>, especially the length of <a
href="https://blog.appdynamics.com/product/estimating-costs-of-storing-documents-in-elasticsearch/";>strings</a>
within your logs (see also <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/5.5/tune-for-disk-usage.html#_use_literal_best_compression_literal";>Elastic’s
tips on indexing strings</a>). One of the discriminating features of Apache
UserALE.js is its precisio [...]
- <div class="language-shell highlighter-rouge"><pre
class="highlight"><code> ...
- <span class="s2">"pageTitle"</span>: <span class="s2">"Discover -
Kibana"</span>,
- <span class="s2">"toolName"</span>: <span class="s2">"test_app"</span>,
- <span class="s2">"userId"</span>: <span class="s2">"nobody"</span>,
- <span class="s2">"type"</span>: <span class="s2">"click"</span>,
- <span class="s2">"target"</span>: <span class="s2">"a.kuiButton
kuiButton--small kuiButton--secondary"</span>,
- <span class="s2">"path"</span>: <span class="o">[</span>
- <span class="s2">"a.kuiButton kuiButton--small
kuiButton--secondary"</span>,
- <span class="s2">"div.kbnDocTableDetails__actions"</span>,
- <span class="s2">"td"</span>,
- <span class="s2">"tr"</span>,
- <span class="s2">"tbody"</span>,
- <span class="s2">"table.kbn-table table"</span>,
- <span class="s2">"div.kbnDocTable__container"</span>,
- <span class="s2">"doc-table"</span>,
- <span class="s2">"section.dscTable"</span>,
- <span class="s2">"div.dscResults"</span>,
- <span class="s2">"div.dscWrapper__content"</span>,
- <span class="s2">"div.dscWrapper col-md-10"</span>,
- <span class="s2">"div.row"</span>,
- <span class="s2">"main.container-fluid"</span>,
- <span class="s2">"discover-app.app-container"</span>,
- <span class="s2">"div.application tab-discover"</span>,
- <span class="s2">"div.app-wrapper-panel"</span>,
- <span class="s2">"div.app-wrapper"</span>,
- <span class="s2">"div.content"</span>,
- <span class="s2">"div#kibana-body"</span>,
- <span class="s2">"body#kibana-app.coreSystemRootDomElement"</span>,
- <span class="s2">"html"</span>,
- <span class="s2">"#document"</span>,
- <span class="s2">"Window"</span>
- ...
+<p>Elasticsearch isn’t a database, its a document store; UserALE.js “logs”
aren’t logs once they’re indexed in Elastic, they become searchable
documents.</p>
+
+<p>Elastic has many useful <a
href="(https://www.elastic.co/blog/found-sizing-elasticsearch)">guides</a> on
sizing and scaling. Below, we’re adding some thoughts based on Apache Flagon’s
own eccentricities.</p>
+
+<p>####Document generation rate is the most important consideration to
scaling</p>
+
+<p>Default <a href="/docs/useralejs">Apache UserALE.js parameters</a> produce
loads of <a href="/docs/useralejs/dataschema">data</a>, even from a single
users.</p>
+
+<p>We strongly suggest that you think about your needs and consider whether
you need data from all our event-handlers. If you don’t need mouseover events,
for example, you can dramatically reduce the rate at which you generate data
and the resources you’ll need.</p>
+
+<p>You can <a href="/docs/useralejs/modifying">modify source</a> or use the <a
href="/docs/useralejs/API">UserALE.js API</a>, and/or use <a
href="/docs/useralejs">configurable HTLM5 parameters in our script tag</a> to
manage data generation rate.</p>
+
+<p>####Resource needs will also grow with <em>document length</em>
+<a
href="https://blog.appdynamics.com/product/estimating-costs-of-storing-documents-in-elasticsearch/";>Strings</a>
within UserALE.js logs (see also <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/5.5/tune-for-disk-usage.html#_use_literal_best_compression_literal";>Elastic’s
tips on indexing strings</a>) can add to scaling needs.</p>
+
+<p>One of the discriminating features of Apache UserALE.js is its precision–it
captures target DOM elements, the DOM path elements are nested in, and loads of
metadata.</p>
+
+<p>UserALE.js fields like <code class="highlighter-rouge">path</code> and
<code class="highlighter-rouge">pageUrl</code> can grow for certain pages. Its
worth considering if you need this level of precision.</p>
+
+<p>Instead, you might consider relying on <code
class="highlighter-rouge">pageTitle</code> rather than <code
class="highlighter-rouge">pageUrl</code>, or just <code
class="highlighter-rouge">target</code> instead of <code
class="highlighter-rouge">path</code></p>
+
+<p>Below is a sample <code class="highlighter-rouge">path</code> for a Kibana
element:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>```shell
+...
+ "pageTitle": "Discover - Kibana",
+ "toolName": "test_app",
+ "userId": "nobody",
+ "type": "click",
+ "target": "a.kuiButton kuiButton--small kuiButton--secondary",
+ "path": [
+ "a.kuiButton kuiButton--small kuiButton--secondary",
+ "div.kbnDocTableDetails__actions",
+ "td",
+ "tr",
+ "tbody",
+ "table.kbn-table table",
+ "div.kbnDocTable__container",
+ "doc-table",
+ "section.dscTable",
+ "div.dscResults",
+ "div.dscWrapper__content",
+ "div.dscWrapper col-md-10",
+ "div.row",
+ "main.container-fluid",
+ "discover-app.app-container",
+ "div.application tab-discover",
+ "div.app-wrapper-panel",
+ "div.app-wrapper",
+ "div.content",
+ "div#kibana-body",
+ "body#kibana-app.coreSystemRootDomElement",
+ "html",
+ "#document",
+ "Window"
+ ...
+```
</code></pre>
- </div>
- </li>
- <li>Apache Flagon is built to scale as a platform, allowing you to connect
additional services to your platform that consume user behavioral data. When
considering scale, the <em>number of services</em> you connect to your Apache
Flagon platform will affect your Elastic stacks’ performance. Any
production-level deployment will require, at minimum a simple three-node <a
href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c";>Elastic
cluster</a> (with one loa [...]
-</ol>
+</div>
+
+<p>Through simple <a href="/docs/useralejs/modifying">modifications to
UserALE.js source</a> or by using the UserALE.js <a
href="/docs/useralejs/API">API</a> you can alias verbose fields in your logs to
reduce resource consumption.</p>
+
+<p>####Additional services attached to your stack can increase resource
consumption</p>
+
+<p>Apache Flagon scales–Elastic products make it easy to attach other services
to your Apache Flagon stack.</p>
+
+<p>The <em>number of services</em> connected to your stack will affect your
Elastic stacks’ performance.</p>
+
+<p>Any production-level deployment will require, at minimum a simple
three-node <a
href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c";>Elastic
cluster</a> (with one load-balancing node).</p>
+
+<p>As you configure that cluster, be mindful that Elasticsearch is indexing
and servicing queries and aggregations for connected servies.</p>
+
+<p>Analytical services connected to the stack can consume significant
resources and increase indexing and search time.</p>
+
+<p>This can be problematic for real-time analytical and monitoring
applications (including Kibana), and result in data loss if logs flush prior to
being indexed.</p>
+
+<p>For hefty analytical services, it may be worth dedicating specific nodes in
your cluster to service them.</p>
-<p>For the reasons above, its really critical to do some benchmarking for your
use-case prior to deciding on a scaling strategy. Below, you’ll find some
guidance for how to do this with Apache Flagon and Elastic tools. We also
provide some simple benchmarks and underlying assumptions. However, each
webpage or application is a bit different, so it’s important create some of
your own benchmarks for comparison with ours.</p>
<h3
id="benchmarking-tools-and-methods-for-sizing-your-apache-flagon-stack">Benchmarking
Tools and Methods for Sizing your Apache Flagon Stack</h3>
-<p>Benchmarking with Elastic isn’t as straight forward as figuring out an
average log size in bytes and then calculating how many logs per second. The
size of Apache UserALE.js logs is too variable (see above) within pages and
across log types. Additionally, a single log may actually be indexed as
separate, nested documents in Elastic. Moreover, Elastic doesn’t really ‘think’
at the document-level, they think about <em>indexes</em>.</p>
+<p>For the reasons above, its really critical to do some benchmarking for your
use-case prior to deciding on a scaling strategy.</p>
+
+<p>This guide outlines a set of tools and steps for running your own
benchmarking study using Flagon’s <a href="/docs/stack">single-node
container</a>.</p>
-<p>This guide outlines a set of tools and steps for running your own
benchmarking study using Flagon’s <a href="/docs/stack">single-node
container</a></p>
+<p>To generate log data, use our <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/example";>UserALE.js
Example</a> test utility or your own website.</p>
-<p>To generate log data, we’ll be using our <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/example";>UserALE.js
Example</a> test kit. This is a useful test utility that makes it easy to <a
href="/docs/useralejs/modifying">modify</a> UserALE.js HTML5 and API parameters
on the fly. Again, for your own purposes, you’ll want to experiment with your
own page/application for more accurate benchmarks.</p>
+<p>Our test utility that makes it easy to <a
href="/docs/useralejs/modifying">modify</a> UserALE.js HTML5 and API parameters
on the fly.</p>
+
+<p>However, you’ll want to experiment with your own page/application for more
accurate benchmarks.</p>
<ol>
<li>
@@ -355,12 +407,12 @@
...
</code></pre>
</div>
- <p>Let’s call this value a baseline for your benchmarking.</p>
+ <p>Let’s call this value a simple benchmark.</p>
- <p>As you continue your benchmarking, the <code
class="highlighter-rouge">userale</code> index “size_in_bytes” will be one of
your key metrics.</p>
+ <p>As you continue benchmarking, the <code
class="highlighter-rouge">userale</code> index “size_in_bytes” will be one of
your key metrics.</p>
</li>
<li>
- <p><strong>Next, let’s see how much data UserALE.js produces with
parameters set to default on your page or app.</strong> We’ll call this:
“drinking from the flame-thrower”.</p>
+ <p><strong>Next, let’s see how much data UserALE.js produces with default
parameters on your page or app.</strong></p>
<p>Drop in a UserALE.js script-tag into your project (see <a
href="/docs/useralejs">instructions</a>).</p>
@@ -374,9 +426,9 @@
></script>
</code></pre>
</div>
- <p>To get a conservative upper-bound, generate as many behaviors in your
page/app as you can (go nuts with mouse-overs, clicks and scrolls).</p>
+ <p>To get a conservative upper-bound, generate as many mouseover and
scroll behaviors in your page/app as you can.</p>
- <p>We did just that for 5 minutes solid. Here’s what our <code
class="highlighter-rouge">userale</code> index looks now (#note
annotations):</p>
+ <p>Doing that for 5 minutes solid, our <code
class="highlighter-rouge">userale</code> index looks like this (#note
annotations):</p>
<div class="language-shell highlighter-rouge"><pre
class="highlight"><code> <span class="s2">"indices"</span> : <span
class="o">{</span>
<span class="s2">"userale"</span> : <span class="o">{</span>
@@ -390,20 +442,24 @@
<span class="s2">"size_in_bytes"</span> : 820978 <span
class="c">#new size of the index (.82 MB)</span>
</code></pre>
</div>
- <p><strong>Some simple math gives +1,998 documents (2000) and +579,866
bytes (.58 MB) generated with UserALE.js by one user in 5 mins</strong>.
Continuing our ultra-conservative benchmarking, lets assume this data
generation rate over an 8 hour period each day, which gives <strong>~56 MB per
day</strong>. At 20 working days in an average month, that gives <strong>1.1 GB
per month</strong>.</p>
+ <p><strong>That’s +1,998 documents (2000) and +579,866 bytes (.58 MB)
generated with UserALE.js by one user in 5 mins</strong>.</p>
+
+ <p>Assuming this rate over an 8 hour period each day for 20 working days:
that’s <strong>1.1 GB per month</strong>.</p>
+
+ <p><strong>This is an ultra-conservative, worst-case-scenario
estimate</strong> because no one uses pages or applications this way.</p>
- <p>Again, <strong>this is ultra-conservative</strong>; no one uses pages
or applications this way (even games). That’s a lot of data for a single-user,
but these are valuable, worst-case-scenario upper-bounds to start with.</p>
+ <p>If you’re a scientist or researcher, these figures might be fine, but
it might be overkill for business analytics.</p>
- <p>If you’re a scientist or researcher, these figures might be fine,
especially if you want to pour over this data. If you’re just interested in how
people are moving through your content, this is probably overkill. The biggest
culprit: take a look at our <code class="highlighter-rouge">Apache Flagon Page
Usage Dashboard</code> to see.</p>
+ <p>To find the biggest culprit in data generation: take a look at our
<code class="highlighter-rouge">Apache Flagon Page Usage Dashboard</code> to
see.</p>
- <p><img
src="https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/mouseOverBench1.png";
alt="alt text" title="Proportion of Mouseovers (Benchmark 1)" /></p>
+ <p><img src="/images/mouseOverBench1.png" width="750" height="500" /></p>
- <p>Mouseovers accounted for a lot of the data we just produced. Raw
mouseover data is written to log very frequently and generates a lot of
documents in Elastic.</p>
+ <p>Mouseovers accounted for a lot of the data we just produced–its written
frequently to index.</p>
</li>
<li>
- <p><strong>Now, let’s take the simplest approach to scaling back
UserALE.js and see how this changes your data generation rate</strong>.</p>
+ <p><strong>Next, scale back UserALE.js mouseover handling and see how this
changes data generation rate</strong>.</p>
- <p>What we’re doing is using the UserALE.js <code
class="highlighter-rouge">settings</code> that can be passed through the script
tag to effectively “downsample” certain channels (e.g., mouseovers) that
generate a lot of documents quickly.</p>
+ <p>Using the UserALE.js HTML5 <code
class="highlighter-rouge">settings</code> in our script tag, you can
“downsample” certain event handler that generate a lot of documents.</p>
<p>Here’s what our script tag looks like now:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <script
@@ -411,7 +467,7 @@
data-url="http://localhost:8100/";
data-user="example-user"
data-version="1.1.1"
- data-resolution=1000 #We've increased the delay between collection of
"high-resolution" events (e.g., mouseover, scrolls) by 100% (default is 500
(ms)).
+ data-resolution=1000 #increased the delay between collection of
"high-resolution" events (e.g., mouseovers).
data-tool="Apache UserALE.js Example"
></script>
</code></pre>
@@ -432,20 +488,24 @@
<span class="s2">"size_in_bytes"</span> : 115978 <span
class="c">#new size of the index (1.1 MB) </span>
</code></pre>
</div>
- <p><strong>Maths give us +1518 documents (500 fewer than our first
benchmark) and +295,000 bytes (.30 MB; ~40% less growth in the store size than
previous) generated by one user in 5 minutes.</strong> That would be
<strong>28.8 MB per day</strong> or <strong>~576 MB per working month</strong>,
per user. We’ve cut down our data generation considerably by modifying one
parameter in our script tag.</p>
+ <p><strong>That is +1518 documents and +295K bytes (.30 MB)</strong></p>
- <p>Why? look at the new proportion of mouseover events… we shaved those
down by ~50%. As that event stream has the highest data generation, net-net we
generated about 25% fewer documents with the same amount and duration of
behavior:</p>
+ <p>But, it’s 500 fewer than our first benchmark and ~40% less growth in
the store.</p>
- <p><img
src="https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/mouseOverBench2.png";
alt="alt text" title="Proportion of Mouseovers (Benchmark 2)" /></p>
+ <p>At <strong>~576 MB per working month</strong> we’ve cut data generation
considerably by modifying one parameter in the script tag.</p>
+
+ <p>The proportion of mouseover events is down by ~50%, and 25% fewer
documents overall:</p>
+
+ <p><img src="/images/mouseOverBench2.png" width="750" height="500" /></p>
</li>
<li>
- <p><strong>Still too much data? Below are some other things that you can
do to curb the growth of your <code class="highlighter-rouge">userale</code>
index</strong> as you continue benchmarking with your page/app.</p>
+ <p><strong>Still too much data? Below are some other ways to curb the
growth of your <code class="highlighter-rouge">userale</code>
index</strong>.</p>
<ul>
- <li>If you don’t need all the different types of events UserALE.js
gathers by default, you can easily curate the list of events you by modifying
<a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> (<code class="highlighter-rouge">attachHandlers.js</code>), then
build a custom UserALE.js script.</li>
- <li>If you don’t need both raw logs (specific records of events) and
interval logs (aggregates of specific events over a time interval), you can
easily drop intervals by modifying UserALE.js <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> (<code class="highlighter-rouge">attachHandlers.js</code> or <code
class="highlighter-rouge">packageLogs.js</code>), then build a custom
UserALE.js script.</li>
- <li>You can reduce the number of meta-data fields written to each
UserALE.js log, by modifying <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> (<code class="highlighter-rouge">packageLogs.js</code>), then build
a custom UserALE.js script.</li>
- <li>If you want different levels of logging granularity (more or less
data) for different pages within your site/app, you can build different
versions of UserALE.js to service different pages. Just name each version
differently and point to the one you want with script-tags, page-by-page.</li>
+ <li>Modify <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> to cut down on event handlers (<code
class="highlighter-rouge">attachHandlers.js</code>).</li>
+ <li>Drop interval logging by modifying UserALE.js <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> (<code class="highlighter-rouge">attachHandlers.js</code> or <code
class="highlighter-rouge">packageLogs.js</code>).</li>
+ <li>Reduce the amount of metadata you collect by modifying <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> (<code class="highlighter-rouge">packageLogs.js</code>).</li>
+ <li>Use different UserALE.js script builds for different pages within
your site/app to serve different logging needs.</li>
<li>You can use our <a href="/docs/useralejs/API">API</a> for surgical
precision in how specific elements (targets) on your page generate data.</li>
</ul>
</li>
@@ -453,84 +513,104 @@
<h3 id="other-tools-to-support-benchmarking-for-scaling">Other Tools to
Support Benchmarking for Scaling</h3>
-<ol>
- <li>
- <p>In our benchmarking guide, we primarily used <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html";>Elastic’s
<code class="highlighter-rouge">Stats</code> API</a>. This provides very
verbose output, which is pretty useful for understanding all of your indexes,
indexing behavior, and how data is being distributed across nodes. You can use
other <a
href="https://www.datadoghq.com/blog/collect-elasticsearch-metrics/#index-stats-api";>Elastic
APIs< [...]
-
-
<p>http://localhost:9200/_cat/indices?format=json&bytes=b&pretty</p>
-
- <p>Output is very simple and index sizes stack on top of one another</p>
- <div class="language-shell highlighter-rouge"><pre
class="highlight"><code> <span class="o">[</span>
- <span class="o">{</span>
- <span class="s2">"health"</span> : <span class="s2">"green"</span>,
- <span class="s2">"status"</span> : <span class="s2">"open"</span>,
- <span class="s2">"index"</span> : <span class="s2">".kibana_1"</span>,
- <span class="s2">"uuid"</span> : <span
class="s2">"FnI_6AQYQEWp2mIxSlM8HQ"</span>,
- <span class="s2">"pri"</span> : <span class="s2">"1"</span>,
- <span class="s2">"rep"</span> : <span class="s2">"0"</span>,
- <span class="s2">"docs.count"</span> : <span class="s2">"184"</span>,
- <span class="s2">"docs.deleted"</span> : <span class="s2">"13"</span>,
- <span class="s2">"store.size"</span> : <span class="s2">"366457"</span>,
- <span class="s2">"pri.store.size"</span> : <span
class="s2">"366457"</span>
- <span class="o">}</span>,
- <span class="o">{</span>
- <span class="s2">"health"</span> : <span class="s2">"yellow"</span>,
- <span class="s2">"status"</span> : <span class="s2">"open"</span>,
- <span class="s2">"index"</span> : <span
class="s2">"metricbeat-6.6.2-2019.04.22"</span>,
- <span class="s2">"uuid"</span> : <span
class="s2">"pDYNmzsxTFu9Z0Tc1_GdLw"</span>,
- <span class="s2">"pri"</span> : <span class="s2">"5"</span>,
- <span class="s2">"rep"</span> : <span class="s2">"1"</span>,
- <span class="s2">"docs.count"</span> : <span class="s2">"4687"</span>,
- <span class="s2">"docs.deleted"</span> : <span class="s2">"0"</span>,
- <span class="s2">"store.size"</span> : <span class="s2">"6309920"</span>,
- <span class="s2">"pri.store.size"</span> : <span
class="s2">"6309920"</span>
- <span class="o">}</span>,
- <span class="o">{</span>
- <span class="s2">"health"</span> : <span class="s2">"green"</span>,
- <span class="s2">"status"</span> : <span class="s2">"open"</span>,
- <span class="s2">"index"</span> : <span class="s2">"userale"</span>,
- <span class="s2">"uuid"</span> : <span
class="s2">"0h0Wxe2cSwqMALs4QCJ8Tw"</span>,
- <span class="s2">"pri"</span> : <span class="s2">"1"</span>,
- <span class="s2">"rep"</span> : <span class="s2">"0"</span>,
- <span class="s2">"docs.count"</span> : <span class="s2">"14018"</span>,
- <span class="s2">"docs.deleted"</span> : <span class="s2">"0"</span>,
- <span class="s2">"store.size"</span> : <span class="s2">"2235963"</span>,
- <span class="s2">"pri.store.size"</span> : <span
class="s2">"2235963"</span>
- <span class="o">}</span>,
- <span class="o">{</span>
- <span class="s2">"health"</span> : <span class="s2">"yellow"</span>,
- <span class="s2">"status"</span> : <span class="s2">"open"</span>,
- <span class="s2">"index"</span> : <span
class="s2">"metricbeat-6.6.2-2019.04.27"</span>,
- <span class="s2">"uuid"</span> : <span
class="s2">"wTyUBXvNRMOwpR4lDF9BNA"</span>,
- <span class="s2">"pri"</span> : <span class="s2">"5"</span>,
- <span class="s2">"rep"</span> : <span class="s2">"1"</span>,
- <span class="s2">"docs.count"</span> : <span class="s2">"107124"</span>,
- <span class="s2">"docs.deleted"</span> : <span class="s2">"0"</span>,
- <span class="s2">"store.size"</span> : <span class="s2">"63263644"</span>,
- <span class="s2">"pri.store.size"</span> : <span
class="s2">"63263644"</span>
- <span class="o">}</span>
- <span class="o">]</span>
+<p>In our benchmarking guide, we primarily used Elastic’s <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html";><code
class="highlighter-rouge">Stats</code> API</a>.</p>
+
+<p>You can <strong>use other <a
href="https://www.datadoghq.com/blog/collect-elasticsearch-metrics/#index-stats-api";>Elastic
APIs</a></strong> for different views of what’s going on inside your Apache
Flagon stack.</p>
+
+<p>For more streamlined views into your indices, try the <code
class="highlighter-rouge">CAT</code> API. Try this call in your browser:</p>
+
+<div class="highlighter-rouge"><pre
class="highlight"><code>http://localhost:9200/_cat/indices?format=json&bytes=b&pretty
+
+Output is very simple and index sizes stack on top of one another
+```shell
+[
+ {
+ "health" : "green",
+ "status" : "open",
+ "index" : ".kibana_1",
+ "uuid" : "FnI_6AQYQEWp2mIxSlM8HQ",
+ "pri" : "1",
+ "rep" : "0",
+ "docs.count" : "184",
+ "docs.deleted" : "13",
+ "store.size" : "366457",
+ "pri.store.size" : "366457"
+ },
+ {
+ "health" : "yellow",
+ "status" : "open",
+ "index" : "metricbeat-6.6.2-2019.04.22",
+ "uuid" : "pDYNmzsxTFu9Z0Tc1_GdLw",
+ "pri" : "5",
+ "rep" : "1",
+ "docs.count" : "4687",
+ "docs.deleted" : "0",
+ "store.size" : "6309920",
+ "pri.store.size" : "6309920"
+ },
+ {
+ "health" : "green",
+ "status" : "open",
+ "index" : "userale",
+ "uuid" : "0h0Wxe2cSwqMALs4QCJ8Tw",
+ "pri" : "1",
+ "rep" : "0",
+ "docs.count" : "14018",
+ "docs.deleted" : "0",
+ "store.size" : "2235963",
+ "pri.store.size" : "2235963"
+ },
+ {
+ "health" : "yellow",
+ "status" : "open",
+ "index" : "metricbeat-6.6.2-2019.04.27",
+ "uuid" : "wTyUBXvNRMOwpR4lDF9BNA",
+ "pri" : "5",
+ "rep" : "1",
+ "docs.count" : "107124",
+ "docs.deleted" : "0",
+ "store.size" : "63263644",
+ "pri.store.size" : "63263644"
+ }
+]
+```
</code></pre>
- </div>
- </li>
- <li>
- <p>There are a lot of other things going on inside your Apache Flagon
Elastic stack. Note above that depending on what you deploy (e.g., metricbeat)
you may have additional indices that need monitoring. For one, Metricbeat
indices grow fast and you’ll want to benchmark that too. The Kibana index will
grow as well, but not as much; it depends on how how many additional objects
you build (e.g., visualizations, indicies, dashboards). Both the <code
class="highlighter-rouge">STATS</code> [...]
- </li>
- <li>
- <p>There are also a lot of things going on inside your server or Docker
containers, depending on how you deploy your Apache Flagon Elastic Stack. This
is why we’ve configured a metricbeat service to run with Flagon. In the same
way that you can benchmark your data generation rate while you generate
behavioral logs in your page/app, you can also look at how your Apache Flagon
stack is utilizing disk and compute resources. When you spin up our single-node
Apache Flagon container, make [...]
+</div>
- <p><img
src="https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/metricBeat.png";
alt="alt text" title="Metricbeat Dashboard" /></p>
- </li>
-</ol>
+<p><strong>Use Flagon’s pre-configured metricbeat service</strong> to run with
Flagon.</p>
+
+<p>You can use this utility to see how your Apache Flagon stack is utilizing
disk and compute resources.</p>
+
+<p>See a sample view of Metricbeat stats below:</p>
+
+<p><img src="/images/metricBeat.png" width="750" height="500" /></p>
<h3 id="wonky-things-that-can-and-will-happen-as-you-benchmark">Wonky Things
that Can and Will Happen as You Benchmark</h3>
-<ol>
- <li>You just finished a benchmarking session after modifying UserALE.js to
produce less data. What you find is that your new store size is either
dramatically bigger than your last benchmark or smaller (which should be
impossible). What happened is a thing called <a
href="https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html";>merging</a>.
It is a good thing, but it can be a confusing thing while you’re benchmarking
against your store size. Essentially, each Ela [...]
-</ol>
+<p>You just finished a benchmarking session after modifying UserALE.js to
produce less data.</p>
+
+<p>What you find is that your new store size is either dramatically bigger
than your last benchmark or smaller (which should be impossible).</p>
+
+<p>What happened is a thing called <a
href="https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html";>merging</a>.</p>
+
+<p>As data is collected it’s gathered into segments within an index.</p>
+
+<p>Each segment is an element of your index and takes up storage, so as your
collect data the number of segments grows quickly consuming a lot of
storage.</p>
+
+<p>As Elastic (Lucene) indexes, it merges these segments into larger and
larger segments, thus reducing the overall number of segments to reduce storage
and resource needs.</p>
+
+<p>This means that depending on when you make a call to Elastic’s <code
class="highlighter-rouge">STATS</code> API, you might be looking at the store
size at different stages in the merging process.</p>
+
+<p>If your store size looks smaller than your last benchmark. You should
re-run it then wait.</p>
+
+<p>If your store size looks way to big, then just wait. After a minute or two,
call the <code class="highlighter-rouge">STATS</code> API again, and you’ll
probably see a store size that makes much more sense.</p>
<h3 id="summary">Summary</h3>
-<p>Benchmarking and adjusting your data-rate so that you can scale how you
want to is made very easy in Apache Flagon. We combine easily deployed and
modified capabilities with the power of Elastic’s APIs and visualization
capabilities. Again, our single-node container is not a scaling solution. It’s
an ingredient, a potent one, that not only serves as the building block for
your cluster, but also a benchmarking tool so that your Apache Flagon cluster
meets your needs for capability, sca [...]
+<p>Benchmarking and adjusting your data-rate so that you can scale how you
want to is made very easy in Apache Flagon.</p>
+
+<p>We combine easily deployed and modified capabilities with the power of
Elastic’s APIs and visualization capabilities.</p>
+
+<p>Again, our single-node container is not a scaling solution. It’s a building
block benchmarking tool to help you build and manage scale and cost.</p>
<p>Subscribe to our <a href="[email protected]">dev
list</a> and join the conversation!</p>
diff --git a/content/docs/useralejs/dataschema/index.html
b/content/docs/useralejs/dataschema/index.html
index ba0482b..a171200 100644
--- a/content/docs/useralejs/dataschema/index.html
+++ b/content/docs/useralejs/dataschema/index.html
@@ -335,7 +335,7 @@ pages. Workflow analysis at this level is really crucial
for understanding:</p>
<p>You may want to specially label some of these fields and important elements
that. See our guide on <a href="/docs/useralejs/API">the UserALE.js API</a>.</p>
<p>For more information about the kinds of behaviors (event classes) we
collect, check out our guide for
-[modifying UserALE.js source code]((/docs/useralejs/modifying).</p>
+<a href="/docs/useralejs/modifying">modifying UserALE.js source code</a>.</p>
<h3 id="temporal-handling-in-useralejs">Temporal Handling in UserALE.js</h3>
diff --git a/content/feed.xml b/content/feed.xml
index 41a1e11..a69525a 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8"?><?xml-stylesheet type="text/xml"
href="http://flagon.incubator.apache.org/feed.xslt.xml";?><feed
xmlns="http://www.w3.org/2005/Atom";><generator uri="http://jekyllrb.com";
version="3.3.1">Jekyll</generator><link
href="http://flagon.incubator.apache.org/feed.xml"; rel="self"
type="application/atom+xml" /><link href="http://flagon.incubator.apache.org/";
rel="alternate" type="text/html"
/><updated>2019-07-02T00:47:19+00:00</updated><id>http://flagon.incubat [...]
+<?xml version="1.0" encoding="utf-8"?><?xml-stylesheet type="text/xml"
href="http://flagon.incubator.apache.org/feed.xslt.xml";?><feed
xmlns="http://www.w3.org/2005/Atom";><generator uri="http://jekyllrb.com";
version="3.3.1">Jekyll</generator><link
href="http://flagon.incubator.apache.org/feed.xml"; rel="self"
type="application/atom+xml" /><link href="http://flagon.incubator.apache.org/";
rel="alternate" type="text/html"
/><updated>2019-07-09T03:56:30+00:00</updated><id>http://flagon.incubat [...]
diff --git a/content/releases/index.html b/content/releases/index.html
index 08247fd..123e115 100644
--- a/content/releases/index.html
+++ b/content/releases/index.html
@@ -67,10 +67,10 @@
Releases
</h1>
<div class="page-content">
- <p>
- Below you can find the official Apache Flagon release distrbution artifacts.
Older releases can be found at the <a
heref="http://archive.apache.org/dist/incubator/flagon/";>Apache Flagon
Archives</a>.
+ <p xmlns="http://www.w3.org/1999/html";>
+ Below you can find the official Apache Flagon release distrbution artifacts.
Older releases can be found at the <a
href="http://archive.apache.org/dist/incubator/flagon/";>Apache Flagon
(Incubating) Archive</a>.
- For ongoing development and future release announcements, stay tuned and
sign up for our <a href="./community.html">mailing lists</a> to keep up to date!
+ For ongoing developments and future release announcements follow our <a
href="https://github.com/apache?q=flagon";>source code</a> and sign up for our
<a href="../community">mailing lists</a> to keep up to date!
</p>
<p>
@@ -100,25 +100,25 @@ The link in the 'Download Artifact' column below should
display a default mirror
<tbody>
<tr>
<td>Apache Flagon UserALE.js 2.0.0 (binary tar.gz)</td>
- <td><a
href="http://www.apache.org/dyn/closer.cgi/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz";>apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz</a></td>
+ <td><a
href="http://www.apache.org/dyn/closer.lua/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz";>apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz.asc";>apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz.asc</a>
</td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz.sha512";>apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz.sha512</a>
</td>
</tr>
<tr>
- <td>Apache Flagon UserALE.js 2.0.0(binary zip)</td>
- <td><a
href="http://www.apache.org/dyn/closer.cgi/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.zip";>apache-flagon-useralejs-incubating-2.0.0-bin.zip</a></td>
+ <td>Apache Flagon UserALE.js 2.0.0 (binary zip)</td>
+ <td><a
href="http://www.apache.org/dyn/closer.lua/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.zip";>apache-flagon-useralejs-incubating-2.0.0-bin.zip</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.zip.asc";>apache-flagon-useralejs-incubating-2.0.0-bin.zip.asc</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.zip.sha512";>apache-flagon-useralejs-incubating-2.0.0-bin.zip.sha512</a></td>
</tr>
<tr>
<td>Apache Flagon UserALE.js 2.0.0 (source tar.gz)</td>
- <td><a
href="http://www.apache.org/dyn/closer.cgi/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.tar.gz";>apache-flagon-useralejs-incubating-2.0.0-src.tar.gz</a></td>
+ <td><a
href="http://www.apache.org/dyn/closer.lua/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.tar.gz";>apache-flagon-useralejs-incubating-2.0.0-src.tar.gz</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.tar.gz.asc";>apache-flagon-useralejs-incubating-2.0.0-src.tar.gz.asc</a>
</td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.tar.gz.sha512";>apache-flagon-useralejs-incubating-2.0.0-src.tar.gz.sha512</a>
</td>
</tr>
<tr>
- <td>Apache Flagon UserALE.js 2.0.0(source zip)</td>
- <td><a
href="http://www.apache.org/dyn/closer.cgi/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.zip";>apache-flagon-useralejs-incubating-2.0.0-src.zip</a></td>
+ <td>Apache Flagon UserALE.js 2.0.0 (source zip)</td>
+ <td><a
href="http://www.apache.org/dyn/closer.lua/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.zip";>apache-flagon-useralejs-incubating-2.0.0-src.zip</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.zip.asc";>apache-flagon-useralejs-incubating-2.0.0-src.zip.asc</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.zip.sha512";>apache-flagon-useralejs-incubating-2.0.0-src.zip.sha512</a></td>
</tr>
diff --git a/site/_docs/stack/scaling.md b/site/_docs/stack/scaling.md
index 6877eaa..0cece36 100644
--- a/site/_docs/stack/scaling.md
+++ b/site/_docs/stack/scaling.md
@@ -7,30 +7,62 @@ priority: 0
### Scaling Apache Flagon: An Introduction and First Principles
-This guide touches on some basic principles to keep in mind as you're planning
for scale with [Apache Flagon](https://github.com/apache/incubator-flagon). We
provide some high-level guidance and considerations for working with an Elastic
stack to scale Flagon that are unique to [Apache UserALE.js]({{
'/docs/useralejs' | prepend: site.baseurl }}) data. We also walk through some
of our benchmarking tools and methodologies, so that you walk into discussions
about scale with some accurate as [...]
+This guide touches on some basic principles to keep in mind as you're planning
for scale with [Apache Flagon](https://github.com/apache/incubator-flagon).
+
+We provide high-level guidance and considerations for working with an Elastic
stack to scale [Apache UserALE.js]({{ '/docs/useralejs' | prepend: site.baseurl
}}) services.
+
+This guide also provides an overview of benchmarking tools and methodologies
for accurately gauging your scaling needs.
**"It Depends..."**
-The best way to scale Apache Flagon depends entirely on your use-case: how
you'll use your Apache UserALE.js data and which [UserALE.js data streams]({{
'/docs/useralejs/dataschema' | prepend: site.baseurl }}) you'll use. This page
provides a few Apache Flagon-specific considerations to think about as you
decide how best to scale, and some useful methods for benchmarking to help you
make that determination.
+The best way to scale Apache Flagon depends entirely on your use-case:
+
+* how you'll use your Apache UserALE.js data;
+* which [data streams]({{ '/docs/useralejs/dataschema' | prepend: site.baseurl
}}) you'll use;
+* how long you need to keep your data.
**The Apache Flagon Single-Node Elastic Container is an Ingredient, Not a
Whole Solution**
-The single-node Elastic (ELK) stack in the Docker container distributed by
[Apache Flagon]({{ '/docs/stack' | prepend: site.baseurl }}) is not alone
suitable for most production-level use-cases. It may be suitable on its own,
for "grab-and-go" user-testing use cases, for example. This would entail just a
few days of data collection from a specific application, from just a few users.
But, alone it will fail quickly for most persistent data collection uses and
enterprise-scale use-cases. R [...]
+The single-node Elastic (ELK) distributed by [Apache Flagon]({{ '/docs/stack'
| prepend: site.baseurl }}) is not alone suitable for most production-level
use-cases.
+
+It may be suitable on its own for "grab-and-go" user-testing use cases; just a
few days of data collection from a specific application, from just a few users.
+
+Alone it will fail quickly for any enterprise-scale use-cases.
+
+Instead, this container is meant to be a building block for larger solutions:
-1. Our ELK .yml config files for our Docker container can be used as the
building-blocks for your very own [multi-node
cluster](https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c)
with load-balancing capabilities.
+1. Our ELK .yml config files for our Docker container can be used as the
building-blocks for your very own load-balanced, [multi-node
cluster](https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c).
1. You can use our [Kubernetes
build](https://github.com/apache/incubator-flagon/tree/master/kubernetes),
which relies on our Docker assets, to scale your Apache Flagon stack to meet
your needs.
1. You can use our single-node container to scale out in [AWS Elastic
Beanstalk (EBS)](https://aws.amazon.com/elasticbeanstalk/).
**Apache Flagon Data Also Scales**
-It's important to note that the burden of scale isn't placed wholly on your
Apache Flagon stack. Flagon's behavioral logging capability, [Apache
UserALE.js]({{ '/docs/useralejs' | prepend: site.baseurl }}) also scales. This
means that one of the most cost-efficient ways to manage the resources behind
your Apache Flagon stack, is to [configure]({{ '/docs/useralejs/API' | prepend:
site.baseurl }}) or [modify]({{ '/docs/useralejs/modifying' | prepend:
site.baseurl }}) UserALE.js to meet you [...]
+Flagon's behavioral logging capability, [Apache UserALE.js]({{
'/docs/useralejs' | prepend: site.baseurl }}) also scales. The most efficient
way to manage scale and resources, is to [configure]({{ '/docs/useralejs/API' |
prepend: site.baseurl }}) or [modify]({{ '/docs/useralejs/modifying' | prepend:
site.baseurl }}) UserALE.js.
### Sizing Up an Elastic Stack
-When thinking about how to scale your Apache Flagon Elastic stack, it's
important to note that Elasticsearch isn't a database, its a datastore, which
stores documents. Elastic is built on top of
[Lucene](http://lucene.apache.org/). That means that Apache Flagon "logs"
aren't logs once they're indexed in Elastic, they become searchable documents.
That's a huge strength (and why we chose Elastic), but it also means that
assumptions about resource consumptions based purely on records and fi [...]
+Elasticsearch isn't a database, its a document store; UserALE.js "logs" aren't
logs once they're indexed in Elastic, they become searchable documents.
+
+Elastic has many useful
[guides]((https://www.elastic.co/blog/found-sizing-elasticsearch)) on sizing
and scaling. Below, we're adding some thoughts based on Apache Flagon's own
eccentricities.
+
+####Document generation rate is the most important consideration to scaling
+
+Default [Apache UserALE.js parameters]({{ '/docs/useralejs' | prepend:
site.baseurl }}) produce loads of [data]({{ '/docs/useralejs/dataschema' |
prepend: site.baseurl }}), even from a single users.
+
+We strongly suggest that you think about your needs and consider whether you
need data from all our event-handlers. If you don't need mouseover events, for
example, you can dramatically reduce the rate at which you generate data and
the resources you'll need.
+
+You can [modify source]({{ '/docs/useralejs/modifying' | prepend: site.baseurl
}}) or use the [UserALE.js API]({{ '/docs/useralejs/API' | prepend:
site.baseurl }}), and/or use [configurable HTLM5 parameters in our script
tag]({{ '/docs/useralejs' | prepend: site.baseurl }}) to manage data generation
rate.
+
+####Resource needs will also grow with *document length*
+[Strings](https://blog.appdynamics.com/product/estimating-costs-of-storing-documents-in-elasticsearch/)
within UserALE.js logs (see also [Elastic's tips on indexing
strings](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/tune-for-disk-usage.html#_use_literal_best_compression_literal))
can add to scaling needs.
-1. Given that documents are the atomic unit of storage in Elastic, *document
generation rate* is the most important consideration to scaling. Default
[Apache UserALE.js parameters]({{ '/docs/useralejs' | prepend: site.baseurl }})
produce a lot of [data]({{ '/docs/useralejs/dataschema' | prepend: site.baseurl
}}), even from single users in Apache UserALE.js. In fact, we used say that
"drinking from the fire-hose" didn't quite do our data-rate justice--we used to
say that opening up UserAL [...]
+One of the discriminating features of Apache UserALE.js is its precision--it
captures target DOM elements, the DOM path elements are nested in, and loads of
metadata.
+
+UserALE.js fields like `path` and `pageUrl` can grow for certain pages. Its
worth considering if you need this level of precision.
+
+Instead, you might consider relying on `pageTitle` rather than `pageUrl`, or
just `target` instead of `path`
+
+Below is a sample `path` for a Kibana element:
-1. Your Elastic resource needs will also grow with *document length*,
especially the length of
[strings](https://blog.appdynamics.com/product/estimating-costs-of-storing-documents-in-elasticsearch/)
within your logs (see also [Elastic's tips on indexing
strings](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/tune-for-disk-usage.html#_use_literal_best_compression_literal)).
One of the discriminating features of Apache UserALE.js is its precision--its
ability to capture both t [...]
```shell
...
"pageTitle": "Discover - Kibana",
@@ -66,17 +98,36 @@ When thinking about how to scale your Apache Flagon Elastic
stack, it's importan
...
```
+Through simple [modifications to UserALE.js source]({{
'/docs/useralejs/modifying' | prepend: site.baseurl }}) or by using the
UserALE.js [API]({{ '/docs/useralejs/API' | prepend: site.baseurl }}) you can
alias verbose fields in your logs to reduce resource consumption.
+
+####Additional services attached to your stack can increase resource
consumption
+
+Apache Flagon scales--Elastic products make it easy to attach other services
to your Apache Flagon stack.
+
+The *number of services* connected to your stack will affect your Elastic
stacks' performance.
+
+Any production-level deployment will require, at minimum a simple three-node
[Elastic
cluster](https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c)
(with one load-balancing node).
+
+As you configure that cluster, be mindful that Elasticsearch is indexing and
servicing queries and aggregations for connected servies.
+
+Analytical services connected to the stack can consume significant resources
and increase indexing and search time.
+
+This can be problematic for real-time analytical and monitoring applications
(including Kibana), and result in data loss if logs flush prior to being
indexed.
+
+For hefty analytical services, it may be worth dedicating specific nodes in
your cluster to service them.
-1. Apache Flagon is built to scale as a platform, allowing you to connect
additional services to your platform that consume user behavioral data. When
considering scale, the *number of services* you connect to your Apache Flagon
platform will affect your Elastic stacks' performance. Any production-level
deployment will require, at minimum a simple three-node [Elastic
cluster](https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c)
(with one load-balancing node). As [...]
-For the reasons above, its really critical to do some benchmarking for your
use-case prior to deciding on a scaling strategy. Below, you'll find some
guidance for how to do this with Apache Flagon and Elastic tools. We also
provide some simple benchmarks and underlying assumptions. However, each
webpage or application is a bit different, so it's important create some of
your own benchmarks for comparison with ours.
### Benchmarking Tools and Methods for Sizing your Apache Flagon Stack
-Benchmarking with Elastic isn't as straight forward as figuring out an average
log size in bytes and then calculating how many logs per second. The size of
Apache UserALE.js logs is too variable (see above) within pages and across log
types. Additionally, a single log may actually be indexed as separate, nested
documents in Elastic. Moreover, Elastic doesn't really 'think' at the
document-level, they think about *indexes*.
-
-This guide outlines a set of tools and steps for running your own benchmarking
study using Flagon's [single-node container]({{ '/docs/stack' | prepend:
site.baseurl }})
+For the reasons above, its really critical to do some benchmarking for your
use-case prior to deciding on a scaling strategy.
-To generate log data, we'll be using our [UserALE.js
Example](https://github.com/apache/incubator-flagon-useralejs/tree/master/example)
test kit. This is a useful test utility that makes it easy to [modify]({{
'/docs/useralejs/modifying' | prepend: site.baseurl }}) UserALE.js HTML5 and
API parameters on the fly. Again, for your own purposes, you'll want to
experiment with your own page/application for more accurate benchmarks.
+This guide outlines a set of tools and steps for running your own benchmarking
study using Flagon's [single-node container]({{ '/docs/stack' | prepend:
site.baseurl }}).
+
+To generate log data, use our [UserALE.js
Example](https://github.com/apache/incubator-flagon-useralejs/tree/master/example)
test utility or your own website.
+
+Our test utility that makes it easy to [modify]({{ '/docs/useralejs/modifying'
| prepend: site.baseurl }}) UserALE.js HTML5 and API parameters on the fly.
+
+However, you'll want to experiment with your own page/application for more
accurate benchmarks.
1. **Start up the Apache Flagon Elastic Stack (detailed instructions
[here](https://github.com/apache/incubator-flagon/tree/master/docker)).**
@@ -108,11 +159,11 @@ To generate log data, we'll be using our [UserALE.js
Example](https://github.com
"size_in_bytes" : 241212 #this is size of the index in bytes
(.24 MB).
...
```
- Let's call this value a baseline for your benchmarking.
+ Let's call this value a simple benchmark.
- As you continue your benchmarking, the `userale` index "size_in_bytes"
will be one of your key metrics.
+ As you continue benchmarking, the `userale` index "size_in_bytes" will be
one of your key metrics.
-1. **Next, let's see how much data UserALE.js produces with parameters set to
default on your page or app.** We'll call this: "drinking from the
flame-thrower".
+1. **Next, let's see how much data UserALE.js produces with default parameters
on your page or app.**
Drop in a UserALE.js script-tag into your project (see [instructions]({{
'/docs/useralejs' | prepend: site.baseurl }})).
@@ -126,9 +177,9 @@ To generate log data, we'll be using our [UserALE.js
Example](https://github.com
data-tool="Apache UserALE.js Example"
></script>
```
- To get a conservative upper-bound, generate as many behaviors in your
page/app as you can (go nuts with mouse-overs, clicks and scrolls).
+ To get a conservative upper-bound, generate as many mouseover and scroll
behaviors in your page/app as you can.
- We did just that for 5 minutes solid. Here's what our `userale` index
looks now (#note annotations):
+ Doing that for 5 minutes solid, our `userale` index looks like this (#note
annotations):
```shell
"indices" : {
@@ -142,19 +193,24 @@ To generate log data, we'll be using our [UserALE.js
Example](https://github.com
"store" : {
"size_in_bytes" : 820978 #new size of the index (.82 MB)
```
- **Some simple math gives +1,998 documents (2000) and +579,866 bytes (.58
MB) generated with UserALE.js by one user in 5 mins**. Continuing our
ultra-conservative benchmarking, lets assume this data generation rate over an
8 hour period each day, which gives **~56 MB per day**. At 20 working days in
an average month, that gives **1.1 GB per month**.
+ **That's +1,998 documents (2000) and +579,866 bytes (.58 MB) generated
with UserALE.js by one user in 5 mins**.
- Again, **this is ultra-conservative**; no one uses pages or applications
this way (even games). That's a lot of data for a single-user, but these are
valuable, worst-case-scenario upper-bounds to start with.
+ Assuming this rate over an 8 hour period each day for 20 working days:
that's **1.1 GB per month**.
- If you're a scientist or researcher, these figures might be fine,
especially if you want to pour over this data. If you're just interested in how
people are moving through your content, this is probably overkill. The biggest
culprit: take a look at our `Apache Flagon Page Usage Dashboard` to see.
+ **This is an ultra-conservative, worst-case-scenario estimate** because no
one uses pages or applications this way.
- ![alt text][mouseOverBench1]
+ If you're a scientist or researcher, these figures might be fine, but it
might be overkill for business analytics.
- Mouseovers accounted for a lot of the data we just produced. Raw mouseover
data is written to log very frequently and generates a lot of documents in
Elastic.
-
-1. **Now, let's take the simplest approach to scaling back UserALE.js and see
how this changes your data generation rate**.
+ To find the biggest culprit in data generation: take a look at our `Apache
Flagon Page Usage Dashboard` to see.
+
+ <img src= "/images/mouseOverBench1.png" width="750" height="500" />
+
+ Mouseovers accounted for a lot of the data we just produced--its written
frequently to index.
+
+
+1. **Next, scale back UserALE.js mouseover handling and see how this changes
data generation rate**.
- What we're doing is using the UserALE.js `settings` that can be passed
through the script tag to effectively "downsample" certain channels (e.g.,
mouseovers) that generate a lot of documents quickly.
+ Using the UserALE.js HTML5 `settings` in our script tag, you can
"downsample" certain event handler that generate a lot of documents.
Here's what our script tag looks like now:
```
@@ -163,7 +219,7 @@ To generate log data, we'll be using our [UserALE.js
Example](https://github.com
data-url="http://localhost:8100/";
data-user="example-user"
data-version="1.1.1"
- data-resolution=1000 #We've increased the delay between collection of
"high-resolution" events (e.g., mouseover, scrolls) by 100% (default is 500
(ms)).
+ data-resolution=1000 #increased the delay between collection of
"high-resolution" events (e.g., mouseovers).
data-tool="Apache UserALE.js Example"
></script>
```
@@ -183,23 +239,31 @@ To generate log data, we'll be using our [UserALE.js
Example](https://github.com
"store" : {
"size_in_bytes" : 115978 #new size of the index (1.1 MB)
```
- **Maths give us +1518 documents (500 fewer than our first benchmark) and
+295,000 bytes (.30 MB; ~40% less growth in the store size than previous)
generated by one user in 5 minutes.** That would be **28.8 MB per day** or
**~576 MB per working month**, per user. We've cut down our data generation
considerably by modifying one parameter in our script tag.
+ **That is +1518 documents and +295K bytes (.30 MB)**
+
+ But, it's 500 fewer than our first benchmark and ~40% less growth in the
store.
+
+ At **~576 MB per working month** we've cut data generation considerably by
modifying one parameter in the script tag.
- Why? look at the new proportion of mouseover events... we shaved those
down by ~50%. As that event stream has the highest data generation, net-net we
generated about 25% fewer documents with the same amount and duration of
behavior:
+ The proportion of mouseover events is down by ~50%, and 25% fewer
documents overall:
- ![alt text][mouseOverBench2]
+ <img src= "/images/mouseOverBench2.png" width="750" height="500" />
-1. **Still too much data? Below are some other things that you can do to curb
the growth of your `userale` index** as you continue benchmarking with your
page/app.
+1. **Still too much data? Below are some other ways to curb the growth of
your `userale` index**.
- * If you don't need all the different types of events UserALE.js gathers
by default, you can easily curate the list of events you by modifying
[UserALE.js
source](https://github.com/apache/incubator-flagon-useralejs/tree/master/src)
(`attachHandlers.js`), then build a custom UserALE.js script.
- * If you don't need both raw logs (specific records of events) and
interval logs (aggregates of specific events over a time interval), you can
easily drop intervals by modifying UserALE.js [UserALE.js
source](https://github.com/apache/incubator-flagon-useralejs/tree/master/src)
(`attachHandlers.js` or `packageLogs.js`), then build a custom UserALE.js
script.
- * You can reduce the number of meta-data fields written to each UserALE.js
log, by modifying [UserALE.js
source](https://github.com/apache/incubator-flagon-useralejs/tree/master/src)
(`packageLogs.js`), then build a custom UserALE.js script.
- * If you want different levels of logging granularity (more or less data)
for different pages within your site/app, you can build different versions of
UserALE.js to service different pages. Just name each version differently and
point to the one you want with script-tags, page-by-page.
+ * Modify [UserALE.js
source](https://github.com/apache/incubator-flagon-useralejs/tree/master/src)
to cut down on event handlers (`attachHandlers.js`).
+ * Drop interval logging by modifying UserALE.js [UserALE.js
source](https://github.com/apache/incubator-flagon-useralejs/tree/master/src)
(`attachHandlers.js` or `packageLogs.js`).
+ * Reduce the amount of metadata you collect by modifying [UserALE.js
source](https://github.com/apache/incubator-flagon-useralejs/tree/master/src)
(`packageLogs.js`).
+ * Use different UserALE.js script builds for different pages within your
site/app to serve different logging needs.
* You can use our [API]({{ '/docs/useralejs/API' | prepend: site.baseurl
}}) for surgical precision in how specific elements (targets) on your page
generate data.
### Other Tools to Support Benchmarking for Scaling
-1. In our benchmarking guide, we primarily used [Elastic's `Stats`
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html).
This provides very verbose output, which is pretty useful for understanding
all of your indexes, indexing behavior, and how data is being distributed
across nodes. You can use other [Elastic
APIs](https://www.datadoghq.com/blog/collect-elasticsearch-metrics/#index-stats-api)
for different views of what's going on inside your Apache F [...]
+In our benchmarking guide, we primarily used Elastic's [`Stats`
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html).
+
+You can **use other [Elastic
APIs](https://www.datadoghq.com/blog/collect-elasticsearch-metrics/#index-stats-api)**
for different views of what's going on inside your Apache Flagon stack.
+
+For more streamlined views into your indices, try the `CAT` API. Try this call
in your browser:
http://localhost:9200/_cat/indices?format=json&bytes=b&pretty
@@ -256,21 +320,41 @@ To generate log data, we'll be using our [UserALE.js
Example](https://github.com
}
]
```
-1. There are a lot of other things going on inside your Apache Flagon Elastic
stack. Note above that depending on what you deploy (e.g., metricbeat) you may
have additional indices that need monitoring. For one, Metricbeat indices grow
fast and you'll want to benchmark that too. The Kibana index will grow as well,
but not as much; it depends on how how many additional objects you build (e.g.,
visualizations, indicies, dashboards). Both the `STATS` and `CAT` APIs are very
useful for seein [...]
+
+**Use Flagon's pre-configured metricbeat service** to run with Flagon.
+
+You can use this utility to see how your Apache Flagon stack is utilizing disk
and compute resources.
-1. There are also a lot of things going on inside your server or Docker
containers, depending on how you deploy your Apache Flagon Elastic Stack. This
is why we've configured a metricbeat service to run with Flagon. In the same
way that you can benchmark your data generation rate while you generate
behavioral logs in your page/app, you can also look at how your Apache Flagon
stack is utilizing disk and compute resources. When you spin up our single-node
Apache Flagon container, make sure [...]
+See a sample view of Metricbeat stats below:
- ![alt text][metricBeat]
+<img src= "/images/metricBeat.png" width="750" height="500" />
### Wonky Things that Can and Will Happen as You Benchmark
-1. You just finished a benchmarking session after modifying UserALE.js to
produce less data. What you find is that your new store size is either
dramatically bigger than your last benchmark or smaller (which should be
impossible). What happened is a thing called
[merging](https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html).
It is a good thing, but it can be a confusing thing while you're benchmarking
against your store size. Essentially, each Elastic shard is [...]
+You just finished a benchmarking session after modifying UserALE.js to produce
less data.
+
+What you find is that your new store size is either dramatically bigger than
your last benchmark or smaller (which should be impossible).
+
+What happened is a thing called
[merging](https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html).
+
+As data is collected it's gathered into segments within an index.
+
+Each segment is an element of your index and takes up storage, so as your
collect data the number of segments grows quickly consuming a lot of storage.
+As Elastic (Lucene) indexes, it merges these segments into larger and larger
segments, thus reducing the overall number of segments to reduce storage and
resource needs.
+
+This means that depending on when you make a call to Elastic's `STATS` API,
you might be looking at the store size at different stages in the merging
process.
+
+If your store size looks smaller than your last benchmark. You should re-run
it then wait.
+
+If your store size looks way to big, then just wait. After a minute or two,
call the `STATS` API again, and you'll probably see a store size that makes
much more sense.
+
### Summary
-Benchmarking and adjusting your data-rate so that you can scale how you want
to is made very easy in Apache Flagon. We combine easily deployed and modified
capabilities with the power of Elastic's APIs and visualization capabilities.
Again, our single-node container is not a scaling solution. It's an ingredient,
a potent one, that not only serves as the building block for your cluster, but
also a benchmarking tool so that your Apache Flagon cluster meets your needs
for capability, scale [...]
+Benchmarking and adjusting your data-rate so that you can scale how you want
to is made very easy in Apache Flagon.
+
+We combine easily deployed and modified capabilities with the power of
Elastic's APIs and visualization capabilities.
+
+Again, our single-node container is not a scaling solution. It's a building
block benchmarking tool to help you build and manage scale and cost.
Subscribe to our [dev list]([email protected]) and
join the conversation!
-[mouseOverBench1]:
https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/mouseOverBench1.png
"Proportion of Mouseovers (Benchmark 1)"
-[mouseOverBench2]:
https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/mouseOverBench2.png
"Proportion of Mouseovers (Benchmark 2)"
-[metricBeat]:
https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/metricBeat.png
"Metricbeat Dashboard"
diff --git a/site/_site/community/index.html b/site/_site/community/index.html
index 6f7099a..75429cf 100644
--- a/site/_site/community/index.html
+++ b/site/_site/community/index.html
@@ -81,7 +81,7 @@
</h3>
<p>
For users of Apache Flagon who want to keep up to date with new releases
and community highlights. To subscribe, email <a
href="mailto:[email protected]";>[email protected]</a>.
- <a class="ui yellow button"
href="mailto:[email protected]";>User List</a>
+ <a class="ui yellow button"
href="mailto:[email protected]";>User List</a>
</p>
<h3 class="ui header">
diff --git a/site/_site/docs/stack/scaling/index.html
b/site/_site/docs/stack/scaling/index.html
index 6b544ab..da8e903 100644
--- a/site/_site/docs/stack/scaling/index.html
+++ b/site/_site/docs/stack/scaling/index.html
@@ -248,80 +248,132 @@
<h2 class="ui header">Scaling Considerations</h2>
<h3
id="scaling-apache-flagon-an-introduction-and-first-principles">Scaling Apache
Flagon: An Introduction and First Principles</h3>
-<p>This guide touches on some basic principles to keep in mind as you’re
planning for scale with <a
href="https://github.com/apache/incubator-flagon";>Apache Flagon</a>. We provide
some high-level guidance and considerations for working with an Elastic stack
to scale Flagon that are unique to <a href="/docs/useralejs">Apache
UserALE.js</a> data. We also walk through some of our benchmarking tools and
methodologies, so that you walk into discussions about scale with some accurate
assumptio [...]
+<p>This guide touches on some basic principles to keep in mind as you’re
planning for scale with <a
href="https://github.com/apache/incubator-flagon";>Apache Flagon</a>.</p>
+
+<p>We provide high-level guidance and considerations for working with an
Elastic stack to scale <a href="/docs/useralejs">Apache UserALE.js</a>
services.</p>
+
+<p>This guide also provides an overview of benchmarking tools and
methodologies for accurately gauging your scaling needs.</p>
<p><strong>“It Depends…“</strong></p>
-<p>The best way to scale Apache Flagon depends entirely on your use-case: how
you’ll use your Apache UserALE.js data and which <a
href="/docs/useralejs/dataschema">UserALE.js data streams</a> you’ll use. This
page provides a few Apache Flagon-specific considerations to think about as you
decide how best to scale, and some useful methods for benchmarking to help you
make that determination.</p>
+<p>The best way to scale Apache Flagon depends entirely on your use-case:</p>
+
+<ul>
+ <li>how you’ll use your Apache UserALE.js data;</li>
+ <li>which <a href="/docs/useralejs/dataschema">data streams</a> you’ll
use;</li>
+ <li>how long you need to keep your data.</li>
+</ul>
<p><strong>The Apache Flagon Single-Node Elastic Container is an Ingredient,
Not a Whole Solution</strong></p>
-<p>The single-node Elastic (ELK) stack in the Docker container distributed by
<a href="/docs/stack">Apache Flagon</a> is not alone suitable for most
production-level use-cases. It may be suitable on its own, for “grab-and-go”
user-testing use cases, for example. This would entail just a few days of data
collection from a specific application, from just a few users. But, alone it
will fail quickly for most persistent data collection uses and enterprise-scale
use-cases. Rather, this contai [...]
+<p>The single-node Elastic (ELK) distributed by <a href="/docs/stack">Apache
Flagon</a> is not alone suitable for most production-level use-cases.</p>
+
+<p>It may be suitable on its own for “grab-and-go” user-testing use cases;
just a few days of data collection from a specific application, from just a few
users.</p>
+
+<p>Alone it will fail quickly for any enterprise-scale use-cases.</p>
+
+<p>Instead, this container is meant to be a building block for larger
solutions:</p>
<ol>
- <li>Our ELK .yml config files for our Docker container can be used as the
building-blocks for your very own <a
href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c";>multi-node
cluster</a> with load-balancing capabilities.</li>
+ <li>Our ELK .yml config files for our Docker container can be used as the
building-blocks for your very own load-balanced, <a
href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c";>multi-node
cluster</a>.</li>
<li>You can use our <a
href="https://github.com/apache/incubator-flagon/tree/master/kubernetes";>Kubernetes
build</a>, which relies on our Docker assets, to scale your Apache Flagon
stack to meet your needs.</li>
<li>You can use our single-node container to scale out in <a
href="https://aws.amazon.com/elasticbeanstalk/";>AWS Elastic Beanstalk
(EBS)</a>.</li>
</ol>
<p><strong>Apache Flagon Data Also Scales</strong></p>
-<p>It’s important to note that the burden of scale isn’t placed wholly on your
Apache Flagon stack. Flagon’s behavioral logging capability, <a
href="/docs/useralejs">Apache UserALE.js</a> also scales. This means that one
of the most cost-efficient ways to manage the resources behind your Apache
Flagon stack, is to <a href="/docs/useralejs/API">configure</a> or <a
href="/docs/useralejs/modifying">modify</a> UserALE.js to meet your
use-case.</p>
+<p>Flagon’s behavioral logging capability, <a href="/docs/useralejs">Apache
UserALE.js</a> also scales. The most efficient way to manage scale and
resources, is to <a href="/docs/useralejs/API">configure</a> or <a
href="/docs/useralejs/modifying">modify</a> UserALE.js.</p>
<h3 id="sizing-up-an-elastic-stack">Sizing Up an Elastic Stack</h3>
-<p>When thinking about how to scale your Apache Flagon Elastic stack, it’s
important to note that Elasticsearch isn’t a database, its a datastore, which
stores documents. Elastic is built on top of <a
href="http://lucene.apache.org/";>Lucene</a>. That means that Apache Flagon
“logs” aren’t logs once they’re indexed in Elastic, they become searchable
documents. That’s a huge strength (and why we chose Elastic), but it also means
that assumptions about resource consumptions based purely on [...]
-
-<ol>
- <li>
- <p>Given that documents are the atomic unit of storage in Elastic,
<em>document generation rate</em> is the most important consideration to
scaling. Default <a href="/docs/useralejs">Apache UserALE.js parameters</a>
produce a lot of <a href="/docs/useralejs/dataschema">data</a>, even from
single users in Apache UserALE.js. In fact, we used say that “drinking from the
fire-hose” didn’t quite do our data-rate justice–we used to say that opening up
UserALE.js was more like “drinking fro [...]
- </li>
- <li>Your Elastic resource needs will also grow with <em>document
length</em>, especially the length of <a
href="https://blog.appdynamics.com/product/estimating-costs-of-storing-documents-in-elasticsearch/";>strings</a>
within your logs (see also <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/5.5/tune-for-disk-usage.html#_use_literal_best_compression_literal";>Elastic’s
tips on indexing strings</a>). One of the discriminating features of Apache
UserALE.js is its precisio [...]
- <div class="language-shell highlighter-rouge"><pre
class="highlight"><code> ...
- <span class="s2">"pageTitle"</span>: <span class="s2">"Discover -
Kibana"</span>,
- <span class="s2">"toolName"</span>: <span class="s2">"test_app"</span>,
- <span class="s2">"userId"</span>: <span class="s2">"nobody"</span>,
- <span class="s2">"type"</span>: <span class="s2">"click"</span>,
- <span class="s2">"target"</span>: <span class="s2">"a.kuiButton
kuiButton--small kuiButton--secondary"</span>,
- <span class="s2">"path"</span>: <span class="o">[</span>
- <span class="s2">"a.kuiButton kuiButton--small
kuiButton--secondary"</span>,
- <span class="s2">"div.kbnDocTableDetails__actions"</span>,
- <span class="s2">"td"</span>,
- <span class="s2">"tr"</span>,
- <span class="s2">"tbody"</span>,
- <span class="s2">"table.kbn-table table"</span>,
- <span class="s2">"div.kbnDocTable__container"</span>,
- <span class="s2">"doc-table"</span>,
- <span class="s2">"section.dscTable"</span>,
- <span class="s2">"div.dscResults"</span>,
- <span class="s2">"div.dscWrapper__content"</span>,
- <span class="s2">"div.dscWrapper col-md-10"</span>,
- <span class="s2">"div.row"</span>,
- <span class="s2">"main.container-fluid"</span>,
- <span class="s2">"discover-app.app-container"</span>,
- <span class="s2">"div.application tab-discover"</span>,
- <span class="s2">"div.app-wrapper-panel"</span>,
- <span class="s2">"div.app-wrapper"</span>,
- <span class="s2">"div.content"</span>,
- <span class="s2">"div#kibana-body"</span>,
- <span class="s2">"body#kibana-app.coreSystemRootDomElement"</span>,
- <span class="s2">"html"</span>,
- <span class="s2">"#document"</span>,
- <span class="s2">"Window"</span>
- ...
+<p>Elasticsearch isn’t a database, its a document store; UserALE.js “logs”
aren’t logs once they’re indexed in Elastic, they become searchable
documents.</p>
+
+<p>Elastic has many useful <a
href="(https://www.elastic.co/blog/found-sizing-elasticsearch)">guides</a> on
sizing and scaling. Below, we’re adding some thoughts based on Apache Flagon’s
own eccentricities.</p>
+
+<p>####Document generation rate is the most important consideration to
scaling</p>
+
+<p>Default <a href="/docs/useralejs">Apache UserALE.js parameters</a> produce
loads of <a href="/docs/useralejs/dataschema">data</a>, even from a single
users.</p>
+
+<p>We strongly suggest that you think about your needs and consider whether
you need data from all our event-handlers. If you don’t need mouseover events,
for example, you can dramatically reduce the rate at which you generate data
and the resources you’ll need.</p>
+
+<p>You can <a href="/docs/useralejs/modifying">modify source</a> or use the <a
href="/docs/useralejs/API">UserALE.js API</a>, and/or use <a
href="/docs/useralejs">configurable HTLM5 parameters in our script tag</a> to
manage data generation rate.</p>
+
+<p>####Resource needs will also grow with <em>document length</em>
+<a
href="https://blog.appdynamics.com/product/estimating-costs-of-storing-documents-in-elasticsearch/";>Strings</a>
within UserALE.js logs (see also <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/5.5/tune-for-disk-usage.html#_use_literal_best_compression_literal";>Elastic’s
tips on indexing strings</a>) can add to scaling needs.</p>
+
+<p>One of the discriminating features of Apache UserALE.js is its precision–it
captures target DOM elements, the DOM path elements are nested in, and loads of
metadata.</p>
+
+<p>UserALE.js fields like <code class="highlighter-rouge">path</code> and
<code class="highlighter-rouge">pageUrl</code> can grow for certain pages. Its
worth considering if you need this level of precision.</p>
+
+<p>Instead, you might consider relying on <code
class="highlighter-rouge">pageTitle</code> rather than <code
class="highlighter-rouge">pageUrl</code>, or just <code
class="highlighter-rouge">target</code> instead of <code
class="highlighter-rouge">path</code></p>
+
+<p>Below is a sample <code class="highlighter-rouge">path</code> for a Kibana
element:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>```shell
+...
+ "pageTitle": "Discover - Kibana",
+ "toolName": "test_app",
+ "userId": "nobody",
+ "type": "click",
+ "target": "a.kuiButton kuiButton--small kuiButton--secondary",
+ "path": [
+ "a.kuiButton kuiButton--small kuiButton--secondary",
+ "div.kbnDocTableDetails__actions",
+ "td",
+ "tr",
+ "tbody",
+ "table.kbn-table table",
+ "div.kbnDocTable__container",
+ "doc-table",
+ "section.dscTable",
+ "div.dscResults",
+ "div.dscWrapper__content",
+ "div.dscWrapper col-md-10",
+ "div.row",
+ "main.container-fluid",
+ "discover-app.app-container",
+ "div.application tab-discover",
+ "div.app-wrapper-panel",
+ "div.app-wrapper",
+ "div.content",
+ "div#kibana-body",
+ "body#kibana-app.coreSystemRootDomElement",
+ "html",
+ "#document",
+ "Window"
+ ...
+```
</code></pre>
- </div>
- </li>
- <li>Apache Flagon is built to scale as a platform, allowing you to connect
additional services to your platform that consume user behavioral data. When
considering scale, the <em>number of services</em> you connect to your Apache
Flagon platform will affect your Elastic stacks’ performance. Any
production-level deployment will require, at minimum a simple three-node <a
href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c";>Elastic
cluster</a> (with one loa [...]
-</ol>
+</div>
+
+<p>Through simple <a href="/docs/useralejs/modifying">modifications to
UserALE.js source</a> or by using the UserALE.js <a
href="/docs/useralejs/API">API</a> you can alias verbose fields in your logs to
reduce resource consumption.</p>
+
+<p>####Additional services attached to your stack can increase resource
consumption</p>
+
+<p>Apache Flagon scales–Elastic products make it easy to attach other services
to your Apache Flagon stack.</p>
+
+<p>The <em>number of services</em> connected to your stack will affect your
Elastic stacks’ performance.</p>
+
+<p>Any production-level deployment will require, at minimum a simple
three-node <a
href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c";>Elastic
cluster</a> (with one load-balancing node).</p>
+
+<p>As you configure that cluster, be mindful that Elasticsearch is indexing
and servicing queries and aggregations for connected servies.</p>
+
+<p>Analytical services connected to the stack can consume significant
resources and increase indexing and search time.</p>
+
+<p>This can be problematic for real-time analytical and monitoring
applications (including Kibana), and result in data loss if logs flush prior to
being indexed.</p>
+
+<p>For hefty analytical services, it may be worth dedicating specific nodes in
your cluster to service them.</p>
-<p>For the reasons above, its really critical to do some benchmarking for your
use-case prior to deciding on a scaling strategy. Below, you’ll find some
guidance for how to do this with Apache Flagon and Elastic tools. We also
provide some simple benchmarks and underlying assumptions. However, each
webpage or application is a bit different, so it’s important create some of
your own benchmarks for comparison with ours.</p>
<h3
id="benchmarking-tools-and-methods-for-sizing-your-apache-flagon-stack">Benchmarking
Tools and Methods for Sizing your Apache Flagon Stack</h3>
-<p>Benchmarking with Elastic isn’t as straight forward as figuring out an
average log size in bytes and then calculating how many logs per second. The
size of Apache UserALE.js logs is too variable (see above) within pages and
across log types. Additionally, a single log may actually be indexed as
separate, nested documents in Elastic. Moreover, Elastic doesn’t really ‘think’
at the document-level, they think about <em>indexes</em>.</p>
+<p>For the reasons above, its really critical to do some benchmarking for your
use-case prior to deciding on a scaling strategy.</p>
+
+<p>This guide outlines a set of tools and steps for running your own
benchmarking study using Flagon’s <a href="/docs/stack">single-node
container</a>.</p>
-<p>This guide outlines a set of tools and steps for running your own
benchmarking study using Flagon’s <a href="/docs/stack">single-node
container</a></p>
+<p>To generate log data, use our <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/example";>UserALE.js
Example</a> test utility or your own website.</p>
-<p>To generate log data, we’ll be using our <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/example";>UserALE.js
Example</a> test kit. This is a useful test utility that makes it easy to <a
href="/docs/useralejs/modifying">modify</a> UserALE.js HTML5 and API parameters
on the fly. Again, for your own purposes, you’ll want to experiment with your
own page/application for more accurate benchmarks.</p>
+<p>Our test utility that makes it easy to <a
href="/docs/useralejs/modifying">modify</a> UserALE.js HTML5 and API parameters
on the fly.</p>
+
+<p>However, you’ll want to experiment with your own page/application for more
accurate benchmarks.</p>
<ol>
<li>
@@ -355,12 +407,12 @@
...
</code></pre>
</div>
- <p>Let’s call this value a baseline for your benchmarking.</p>
+ <p>Let’s call this value a simple benchmark.</p>
- <p>As you continue your benchmarking, the <code
class="highlighter-rouge">userale</code> index “size_in_bytes” will be one of
your key metrics.</p>
+ <p>As you continue benchmarking, the <code
class="highlighter-rouge">userale</code> index “size_in_bytes” will be one of
your key metrics.</p>
</li>
<li>
- <p><strong>Next, let’s see how much data UserALE.js produces with
parameters set to default on your page or app.</strong> We’ll call this:
“drinking from the flame-thrower”.</p>
+ <p><strong>Next, let’s see how much data UserALE.js produces with default
parameters on your page or app.</strong></p>
<p>Drop in a UserALE.js script-tag into your project (see <a
href="/docs/useralejs">instructions</a>).</p>
@@ -374,9 +426,9 @@
></script>
</code></pre>
</div>
- <p>To get a conservative upper-bound, generate as many behaviors in your
page/app as you can (go nuts with mouse-overs, clicks and scrolls).</p>
+ <p>To get a conservative upper-bound, generate as many mouseover and
scroll behaviors in your page/app as you can.</p>
- <p>We did just that for 5 minutes solid. Here’s what our <code
class="highlighter-rouge">userale</code> index looks now (#note
annotations):</p>
+ <p>Doing that for 5 minutes solid, our <code
class="highlighter-rouge">userale</code> index looks like this (#note
annotations):</p>
<div class="language-shell highlighter-rouge"><pre
class="highlight"><code> <span class="s2">"indices"</span> : <span
class="o">{</span>
<span class="s2">"userale"</span> : <span class="o">{</span>
@@ -390,20 +442,24 @@
<span class="s2">"size_in_bytes"</span> : 820978 <span
class="c">#new size of the index (.82 MB)</span>
</code></pre>
</div>
- <p><strong>Some simple math gives +1,998 documents (2000) and +579,866
bytes (.58 MB) generated with UserALE.js by one user in 5 mins</strong>.
Continuing our ultra-conservative benchmarking, lets assume this data
generation rate over an 8 hour period each day, which gives <strong>~56 MB per
day</strong>. At 20 working days in an average month, that gives <strong>1.1 GB
per month</strong>.</p>
+ <p><strong>That’s +1,998 documents (2000) and +579,866 bytes (.58 MB)
generated with UserALE.js by one user in 5 mins</strong>.</p>
+
+ <p>Assuming this rate over an 8 hour period each day for 20 working days:
that’s <strong>1.1 GB per month</strong>.</p>
+
+ <p><strong>This is an ultra-conservative, worst-case-scenario
estimate</strong> because no one uses pages or applications this way.</p>
- <p>Again, <strong>this is ultra-conservative</strong>; no one uses pages
or applications this way (even games). That’s a lot of data for a single-user,
but these are valuable, worst-case-scenario upper-bounds to start with.</p>
+ <p>If you’re a scientist or researcher, these figures might be fine, but
it might be overkill for business analytics.</p>
- <p>If you’re a scientist or researcher, these figures might be fine,
especially if you want to pour over this data. If you’re just interested in how
people are moving through your content, this is probably overkill. The biggest
culprit: take a look at our <code class="highlighter-rouge">Apache Flagon Page
Usage Dashboard</code> to see.</p>
+ <p>To find the biggest culprit in data generation: take a look at our
<code class="highlighter-rouge">Apache Flagon Page Usage Dashboard</code> to
see.</p>
- <p><img
src="https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/mouseOverBench1.png";
alt="alt text" title="Proportion of Mouseovers (Benchmark 1)" /></p>
+ <p><img src="/images/mouseOverBench1.png" width="750" height="500" /></p>
- <p>Mouseovers accounted for a lot of the data we just produced. Raw
mouseover data is written to log very frequently and generates a lot of
documents in Elastic.</p>
+ <p>Mouseovers accounted for a lot of the data we just produced–its written
frequently to index.</p>
</li>
<li>
- <p><strong>Now, let’s take the simplest approach to scaling back
UserALE.js and see how this changes your data generation rate</strong>.</p>
+ <p><strong>Next, scale back UserALE.js mouseover handling and see how this
changes data generation rate</strong>.</p>
- <p>What we’re doing is using the UserALE.js <code
class="highlighter-rouge">settings</code> that can be passed through the script
tag to effectively “downsample” certain channels (e.g., mouseovers) that
generate a lot of documents quickly.</p>
+ <p>Using the UserALE.js HTML5 <code
class="highlighter-rouge">settings</code> in our script tag, you can
“downsample” certain event handler that generate a lot of documents.</p>
<p>Here’s what our script tag looks like now:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <script
@@ -411,7 +467,7 @@
data-url="http://localhost:8100/";
data-user="example-user"
data-version="1.1.1"
- data-resolution=1000 #We've increased the delay between collection of
"high-resolution" events (e.g., mouseover, scrolls) by 100% (default is 500
(ms)).
+ data-resolution=1000 #increased the delay between collection of
"high-resolution" events (e.g., mouseovers).
data-tool="Apache UserALE.js Example"
></script>
</code></pre>
@@ -432,20 +488,24 @@
<span class="s2">"size_in_bytes"</span> : 115978 <span
class="c">#new size of the index (1.1 MB) </span>
</code></pre>
</div>
- <p><strong>Maths give us +1518 documents (500 fewer than our first
benchmark) and +295,000 bytes (.30 MB; ~40% less growth in the store size than
previous) generated by one user in 5 minutes.</strong> That would be
<strong>28.8 MB per day</strong> or <strong>~576 MB per working month</strong>,
per user. We’ve cut down our data generation considerably by modifying one
parameter in our script tag.</p>
+ <p><strong>That is +1518 documents and +295K bytes (.30 MB)</strong></p>
- <p>Why? look at the new proportion of mouseover events… we shaved those
down by ~50%. As that event stream has the highest data generation, net-net we
generated about 25% fewer documents with the same amount and duration of
behavior:</p>
+ <p>But, it’s 500 fewer than our first benchmark and ~40% less growth in
the store.</p>
- <p><img
src="https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/mouseOverBench2.png";
alt="alt text" title="Proportion of Mouseovers (Benchmark 2)" /></p>
+ <p>At <strong>~576 MB per working month</strong> we’ve cut data generation
considerably by modifying one parameter in the script tag.</p>
+
+ <p>The proportion of mouseover events is down by ~50%, and 25% fewer
documents overall:</p>
+
+ <p><img src="/images/mouseOverBench2.png" width="750" height="500" /></p>
</li>
<li>
- <p><strong>Still too much data? Below are some other things that you can
do to curb the growth of your <code class="highlighter-rouge">userale</code>
index</strong> as you continue benchmarking with your page/app.</p>
+ <p><strong>Still too much data? Below are some other ways to curb the
growth of your <code class="highlighter-rouge">userale</code>
index</strong>.</p>
<ul>
- <li>If you don’t need all the different types of events UserALE.js
gathers by default, you can easily curate the list of events you by modifying
<a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> (<code class="highlighter-rouge">attachHandlers.js</code>), then
build a custom UserALE.js script.</li>
- <li>If you don’t need both raw logs (specific records of events) and
interval logs (aggregates of specific events over a time interval), you can
easily drop intervals by modifying UserALE.js <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> (<code class="highlighter-rouge">attachHandlers.js</code> or <code
class="highlighter-rouge">packageLogs.js</code>), then build a custom
UserALE.js script.</li>
- <li>You can reduce the number of meta-data fields written to each
UserALE.js log, by modifying <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> (<code class="highlighter-rouge">packageLogs.js</code>), then build
a custom UserALE.js script.</li>
- <li>If you want different levels of logging granularity (more or less
data) for different pages within your site/app, you can build different
versions of UserALE.js to service different pages. Just name each version
differently and point to the one you want with script-tags, page-by-page.</li>
+ <li>Modify <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> to cut down on event handlers (<code
class="highlighter-rouge">attachHandlers.js</code>).</li>
+ <li>Drop interval logging by modifying UserALE.js <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> (<code class="highlighter-rouge">attachHandlers.js</code> or <code
class="highlighter-rouge">packageLogs.js</code>).</li>
+ <li>Reduce the amount of metadata you collect by modifying <a
href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src";>UserALE.js
source</a> (<code class="highlighter-rouge">packageLogs.js</code>).</li>
+ <li>Use different UserALE.js script builds for different pages within
your site/app to serve different logging needs.</li>
<li>You can use our <a href="/docs/useralejs/API">API</a> for surgical
precision in how specific elements (targets) on your page generate data.</li>
</ul>
</li>
@@ -453,84 +513,104 @@
<h3 id="other-tools-to-support-benchmarking-for-scaling">Other Tools to
Support Benchmarking for Scaling</h3>
-<ol>
- <li>
- <p>In our benchmarking guide, we primarily used <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html";>Elastic’s
<code class="highlighter-rouge">Stats</code> API</a>. This provides very
verbose output, which is pretty useful for understanding all of your indexes,
indexing behavior, and how data is being distributed across nodes. You can use
other <a
href="https://www.datadoghq.com/blog/collect-elasticsearch-metrics/#index-stats-api";>Elastic
APIs< [...]
-
-
<p>http://localhost:9200/_cat/indices?format=json&bytes=b&pretty</p>
-
- <p>Output is very simple and index sizes stack on top of one another</p>
- <div class="language-shell highlighter-rouge"><pre
class="highlight"><code> <span class="o">[</span>
- <span class="o">{</span>
- <span class="s2">"health"</span> : <span class="s2">"green"</span>,
- <span class="s2">"status"</span> : <span class="s2">"open"</span>,
- <span class="s2">"index"</span> : <span class="s2">".kibana_1"</span>,
- <span class="s2">"uuid"</span> : <span
class="s2">"FnI_6AQYQEWp2mIxSlM8HQ"</span>,
- <span class="s2">"pri"</span> : <span class="s2">"1"</span>,
- <span class="s2">"rep"</span> : <span class="s2">"0"</span>,
- <span class="s2">"docs.count"</span> : <span class="s2">"184"</span>,
- <span class="s2">"docs.deleted"</span> : <span class="s2">"13"</span>,
- <span class="s2">"store.size"</span> : <span class="s2">"366457"</span>,
- <span class="s2">"pri.store.size"</span> : <span
class="s2">"366457"</span>
- <span class="o">}</span>,
- <span class="o">{</span>
- <span class="s2">"health"</span> : <span class="s2">"yellow"</span>,
- <span class="s2">"status"</span> : <span class="s2">"open"</span>,
- <span class="s2">"index"</span> : <span
class="s2">"metricbeat-6.6.2-2019.04.22"</span>,
- <span class="s2">"uuid"</span> : <span
class="s2">"pDYNmzsxTFu9Z0Tc1_GdLw"</span>,
- <span class="s2">"pri"</span> : <span class="s2">"5"</span>,
- <span class="s2">"rep"</span> : <span class="s2">"1"</span>,
- <span class="s2">"docs.count"</span> : <span class="s2">"4687"</span>,
- <span class="s2">"docs.deleted"</span> : <span class="s2">"0"</span>,
- <span class="s2">"store.size"</span> : <span class="s2">"6309920"</span>,
- <span class="s2">"pri.store.size"</span> : <span
class="s2">"6309920"</span>
- <span class="o">}</span>,
- <span class="o">{</span>
- <span class="s2">"health"</span> : <span class="s2">"green"</span>,
- <span class="s2">"status"</span> : <span class="s2">"open"</span>,
- <span class="s2">"index"</span> : <span class="s2">"userale"</span>,
- <span class="s2">"uuid"</span> : <span
class="s2">"0h0Wxe2cSwqMALs4QCJ8Tw"</span>,
- <span class="s2">"pri"</span> : <span class="s2">"1"</span>,
- <span class="s2">"rep"</span> : <span class="s2">"0"</span>,
- <span class="s2">"docs.count"</span> : <span class="s2">"14018"</span>,
- <span class="s2">"docs.deleted"</span> : <span class="s2">"0"</span>,
- <span class="s2">"store.size"</span> : <span class="s2">"2235963"</span>,
- <span class="s2">"pri.store.size"</span> : <span
class="s2">"2235963"</span>
- <span class="o">}</span>,
- <span class="o">{</span>
- <span class="s2">"health"</span> : <span class="s2">"yellow"</span>,
- <span class="s2">"status"</span> : <span class="s2">"open"</span>,
- <span class="s2">"index"</span> : <span
class="s2">"metricbeat-6.6.2-2019.04.27"</span>,
- <span class="s2">"uuid"</span> : <span
class="s2">"wTyUBXvNRMOwpR4lDF9BNA"</span>,
- <span class="s2">"pri"</span> : <span class="s2">"5"</span>,
- <span class="s2">"rep"</span> : <span class="s2">"1"</span>,
- <span class="s2">"docs.count"</span> : <span class="s2">"107124"</span>,
- <span class="s2">"docs.deleted"</span> : <span class="s2">"0"</span>,
- <span class="s2">"store.size"</span> : <span class="s2">"63263644"</span>,
- <span class="s2">"pri.store.size"</span> : <span
class="s2">"63263644"</span>
- <span class="o">}</span>
- <span class="o">]</span>
+<p>In our benchmarking guide, we primarily used Elastic’s <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html";><code
class="highlighter-rouge">Stats</code> API</a>.</p>
+
+<p>You can <strong>use other <a
href="https://www.datadoghq.com/blog/collect-elasticsearch-metrics/#index-stats-api";>Elastic
APIs</a></strong> for different views of what’s going on inside your Apache
Flagon stack.</p>
+
+<p>For more streamlined views into your indices, try the <code
class="highlighter-rouge">CAT</code> API. Try this call in your browser:</p>
+
+<div class="highlighter-rouge"><pre
class="highlight"><code>http://localhost:9200/_cat/indices?format=json&bytes=b&pretty
+
+Output is very simple and index sizes stack on top of one another
+```shell
+[
+ {
+ "health" : "green",
+ "status" : "open",
+ "index" : ".kibana_1",
+ "uuid" : "FnI_6AQYQEWp2mIxSlM8HQ",
+ "pri" : "1",
+ "rep" : "0",
+ "docs.count" : "184",
+ "docs.deleted" : "13",
+ "store.size" : "366457",
+ "pri.store.size" : "366457"
+ },
+ {
+ "health" : "yellow",
+ "status" : "open",
+ "index" : "metricbeat-6.6.2-2019.04.22",
+ "uuid" : "pDYNmzsxTFu9Z0Tc1_GdLw",
+ "pri" : "5",
+ "rep" : "1",
+ "docs.count" : "4687",
+ "docs.deleted" : "0",
+ "store.size" : "6309920",
+ "pri.store.size" : "6309920"
+ },
+ {
+ "health" : "green",
+ "status" : "open",
+ "index" : "userale",
+ "uuid" : "0h0Wxe2cSwqMALs4QCJ8Tw",
+ "pri" : "1",
+ "rep" : "0",
+ "docs.count" : "14018",
+ "docs.deleted" : "0",
+ "store.size" : "2235963",
+ "pri.store.size" : "2235963"
+ },
+ {
+ "health" : "yellow",
+ "status" : "open",
+ "index" : "metricbeat-6.6.2-2019.04.27",
+ "uuid" : "wTyUBXvNRMOwpR4lDF9BNA",
+ "pri" : "5",
+ "rep" : "1",
+ "docs.count" : "107124",
+ "docs.deleted" : "0",
+ "store.size" : "63263644",
+ "pri.store.size" : "63263644"
+ }
+]
+```
</code></pre>
- </div>
- </li>
- <li>
- <p>There are a lot of other things going on inside your Apache Flagon
Elastic stack. Note above that depending on what you deploy (e.g., metricbeat)
you may have additional indices that need monitoring. For one, Metricbeat
indices grow fast and you’ll want to benchmark that too. The Kibana index will
grow as well, but not as much; it depends on how how many additional objects
you build (e.g., visualizations, indicies, dashboards). Both the <code
class="highlighter-rouge">STATS</code> [...]
- </li>
- <li>
- <p>There are also a lot of things going on inside your server or Docker
containers, depending on how you deploy your Apache Flagon Elastic Stack. This
is why we’ve configured a metricbeat service to run with Flagon. In the same
way that you can benchmark your data generation rate while you generate
behavioral logs in your page/app, you can also look at how your Apache Flagon
stack is utilizing disk and compute resources. When you spin up our single-node
Apache Flagon container, make [...]
+</div>
- <p><img
src="https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/metricBeat.png";
alt="alt text" title="Metricbeat Dashboard" /></p>
- </li>
-</ol>
+<p><strong>Use Flagon’s pre-configured metricbeat service</strong> to run with
Flagon.</p>
+
+<p>You can use this utility to see how your Apache Flagon stack is utilizing
disk and compute resources.</p>
+
+<p>See a sample view of Metricbeat stats below:</p>
+
+<p><img src="/images/metricBeat.png" width="750" height="500" /></p>
<h3 id="wonky-things-that-can-and-will-happen-as-you-benchmark">Wonky Things
that Can and Will Happen as You Benchmark</h3>
-<ol>
- <li>You just finished a benchmarking session after modifying UserALE.js to
produce less data. What you find is that your new store size is either
dramatically bigger than your last benchmark or smaller (which should be
impossible). What happened is a thing called <a
href="https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html";>merging</a>.
It is a good thing, but it can be a confusing thing while you’re benchmarking
against your store size. Essentially, each Ela [...]
-</ol>
+<p>You just finished a benchmarking session after modifying UserALE.js to
produce less data.</p>
+
+<p>What you find is that your new store size is either dramatically bigger
than your last benchmark or smaller (which should be impossible).</p>
+
+<p>What happened is a thing called <a
href="https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html";>merging</a>.</p>
+
+<p>As data is collected it’s gathered into segments within an index.</p>
+
+<p>Each segment is an element of your index and takes up storage, so as your
collect data the number of segments grows quickly consuming a lot of
storage.</p>
+
+<p>As Elastic (Lucene) indexes, it merges these segments into larger and
larger segments, thus reducing the overall number of segments to reduce storage
and resource needs.</p>
+
+<p>This means that depending on when you make a call to Elastic’s <code
class="highlighter-rouge">STATS</code> API, you might be looking at the store
size at different stages in the merging process.</p>
+
+<p>If your store size looks smaller than your last benchmark. You should
re-run it then wait.</p>
+
+<p>If your store size looks way to big, then just wait. After a minute or two,
call the <code class="highlighter-rouge">STATS</code> API again, and you’ll
probably see a store size that makes much more sense.</p>
<h3 id="summary">Summary</h3>
-<p>Benchmarking and adjusting your data-rate so that you can scale how you
want to is made very easy in Apache Flagon. We combine easily deployed and
modified capabilities with the power of Elastic’s APIs and visualization
capabilities. Again, our single-node container is not a scaling solution. It’s
an ingredient, a potent one, that not only serves as the building block for
your cluster, but also a benchmarking tool so that your Apache Flagon cluster
meets your needs for capability, sca [...]
+<p>Benchmarking and adjusting your data-rate so that you can scale how you
want to is made very easy in Apache Flagon.</p>
+
+<p>We combine easily deployed and modified capabilities with the power of
Elastic’s APIs and visualization capabilities.</p>
+
+<p>Again, our single-node container is not a scaling solution. It’s a building
block benchmarking tool to help you build and manage scale and cost.</p>
<p>Subscribe to our <a href="[email protected]">dev
list</a> and join the conversation!</p>
diff --git a/site/_site/docs/useralejs/dataschema/index.html
b/site/_site/docs/useralejs/dataschema/index.html
index ba0482b..a171200 100644
--- a/site/_site/docs/useralejs/dataschema/index.html
+++ b/site/_site/docs/useralejs/dataschema/index.html
@@ -335,7 +335,7 @@ pages. Workflow analysis at this level is really crucial
for understanding:</p>
<p>You may want to specially label some of these fields and important elements
that. See our guide on <a href="/docs/useralejs/API">the UserALE.js API</a>.</p>
<p>For more information about the kinds of behaviors (event classes) we
collect, check out our guide for
-[modifying UserALE.js source code]((/docs/useralejs/modifying).</p>
+<a href="/docs/useralejs/modifying">modifying UserALE.js source code</a>.</p>
<h3 id="temporal-handling-in-useralejs">Temporal Handling in UserALE.js</h3>
diff --git a/site/_site/feed.xml b/site/_site/feed.xml
index 41a1e11..a69525a 100644
--- a/site/_site/feed.xml
+++ b/site/_site/feed.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8"?><?xml-stylesheet type="text/xml"
href="http://flagon.incubator.apache.org/feed.xslt.xml";?><feed
xmlns="http://www.w3.org/2005/Atom";><generator uri="http://jekyllrb.com";
version="3.3.1">Jekyll</generator><link
href="http://flagon.incubator.apache.org/feed.xml"; rel="self"
type="application/atom+xml" /><link href="http://flagon.incubator.apache.org/";
rel="alternate" type="text/html"
/><updated>2019-07-02T00:47:19+00:00</updated><id>http://flagon.incubat [...]
+<?xml version="1.0" encoding="utf-8"?><?xml-stylesheet type="text/xml"
href="http://flagon.incubator.apache.org/feed.xslt.xml";?><feed
xmlns="http://www.w3.org/2005/Atom";><generator uri="http://jekyllrb.com";
version="3.3.1">Jekyll</generator><link
href="http://flagon.incubator.apache.org/feed.xml"; rel="self"
type="application/atom+xml" /><link href="http://flagon.incubator.apache.org/";
rel="alternate" type="text/html"
/><updated>2019-07-09T03:56:30+00:00</updated><id>http://flagon.incubat [...]
diff --git a/site/_site/releases/index.html b/site/_site/releases/index.html
index 08247fd..123e115 100644
--- a/site/_site/releases/index.html
+++ b/site/_site/releases/index.html
@@ -67,10 +67,10 @@
Releases
</h1>
<div class="page-content">
- <p>
- Below you can find the official Apache Flagon release distrbution artifacts.
Older releases can be found at the <a
heref="http://archive.apache.org/dist/incubator/flagon/";>Apache Flagon
Archives</a>.
+ <p xmlns="http://www.w3.org/1999/html";>
+ Below you can find the official Apache Flagon release distrbution artifacts.
Older releases can be found at the <a
href="http://archive.apache.org/dist/incubator/flagon/";>Apache Flagon
(Incubating) Archive</a>.
- For ongoing development and future release announcements, stay tuned and
sign up for our <a href="./community.html">mailing lists</a> to keep up to date!
+ For ongoing developments and future release announcements follow our <a
href="https://github.com/apache?q=flagon";>source code</a> and sign up for our
<a href="../community">mailing lists</a> to keep up to date!
</p>
<p>
@@ -100,25 +100,25 @@ The link in the 'Download Artifact' column below should
display a default mirror
<tbody>
<tr>
<td>Apache Flagon UserALE.js 2.0.0 (binary tar.gz)</td>
- <td><a
href="http://www.apache.org/dyn/closer.cgi/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz";>apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz</a></td>
+ <td><a
href="http://www.apache.org/dyn/closer.lua/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz";>apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz.asc";>apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz.asc</a>
</td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz.sha512";>apache-flagon-useralejs-incubating-2.0.0-bin.tar.gz.sha512</a>
</td>
</tr>
<tr>
- <td>Apache Flagon UserALE.js 2.0.0(binary zip)</td>
- <td><a
href="http://www.apache.org/dyn/closer.cgi/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.zip";>apache-flagon-useralejs-incubating-2.0.0-bin.zip</a></td>
+ <td>Apache Flagon UserALE.js 2.0.0 (binary zip)</td>
+ <td><a
href="http://www.apache.org/dyn/closer.lua/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.zip";>apache-flagon-useralejs-incubating-2.0.0-bin.zip</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.zip.asc";>apache-flagon-useralejs-incubating-2.0.0-bin.zip.asc</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-bin.zip.sha512";>apache-flagon-useralejs-incubating-2.0.0-bin.zip.sha512</a></td>
</tr>
<tr>
<td>Apache Flagon UserALE.js 2.0.0 (source tar.gz)</td>
- <td><a
href="http://www.apache.org/dyn/closer.cgi/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.tar.gz";>apache-flagon-useralejs-incubating-2.0.0-src.tar.gz</a></td>
+ <td><a
href="http://www.apache.org/dyn/closer.lua/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.tar.gz";>apache-flagon-useralejs-incubating-2.0.0-src.tar.gz</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.tar.gz.asc";>apache-flagon-useralejs-incubating-2.0.0-src.tar.gz.asc</a>
</td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.tar.gz.sha512";>apache-flagon-useralejs-incubating-2.0.0-src.tar.gz.sha512</a>
</td>
</tr>
<tr>
- <td>Apache Flagon UserALE.js 2.0.0(source zip)</td>
- <td><a
href="http://www.apache.org/dyn/closer.cgi/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.zip";>apache-flagon-useralejs-incubating-2.0.0-src.zip</a></td>
+ <td>Apache Flagon UserALE.js 2.0.0 (source zip)</td>
+ <td><a
href="http://www.apache.org/dyn/closer.lua/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.zip";>apache-flagon-useralejs-incubating-2.0.0-src.zip</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.zip.asc";>apache-flagon-useralejs-incubating-2.0.0-src.zip.asc</a></td>
<td><a
href="https://dist.apache.org/repos/dist/release/incubator/flagon/apache-flagon-useralejs-incubating-2.0.0/apache-flagon-useralejs-incubating-2.0.0-src.zip.sha512";>apache-flagon-useralejs-incubating-2.0.0-src.zip.sha512</a></td>
</tr>
![https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/mouseOverBench1.png"](https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/mouseOverBench1.png"){kind=link}
![https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/mouseOverBench2.png"](https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/mouseOverBench2.png"){kind=link}
![https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/metricBeat.png"](https://github.com/apache/incubator-flagon/blob/FLAGON-344/site/_site/images/metricBeat.png"){kind=link}
