This is an automated email from the ASF dual-hosted git repository.
poorejc pushed a commit to branch FLAGON-344
in repository https://gitbox.apache.org/repos/asf/incubator-flagon.git
The following commit(s) were added to refs/heads/FLAGON-344 by this push:
new 74a7ce4 [FLAGON-379] Finished Scaling guide (just need to add image
references), made minor related changes to other docs referenced in scaling
guide
74a7ce4 is described below
commit 74a7ce4685a9c785dd8f44111fd825a7a9b3bc43
Author: poorejc <[email protected]>
AuthorDate: Sat Apr 27 14:18:13 2019 -0400
[FLAGON-379] Finished Scaling guide (just need to add image references),
made minor related changes to other docs referenced in scaling guide
---
site/_docs/stack/index.md | 2 +-
site/_docs/stack/scaling.md | 167 ++++++++++++++++++++++++++++++++++++++++--
site/_docs/useralejs/index.md | 2 +-
3 files changed, 161 insertions(+), 10 deletions(-)
diff --git a/site/_docs/stack/index.md b/site/_docs/stack/index.md
index ffc943f..c9a4918 100644
--- a/site/_docs/stack/index.md
+++ b/site/_docs/stack/index.md
@@ -27,7 +27,7 @@ Then, **build UserALE.js**.
$ npm run build
```
-The build process produced a minified version of UserALE.js and a Web
Extension package, giving you two options depending on your needs.
+The build process produces a minified version of UserALE.js and a Web
Extension package, giving you two options depending on your needs. You can skip
the build process if you just want to explore UserALE.js; just use the minified
script found in our
[repo](https://github.com/apache/incubator-flagon-useralejs/tree/master/build).
**Option 1: Include Apache UserALE.js in your project:**
diff --git a/site/_docs/stack/scaling.md b/site/_docs/stack/scaling.md
index 719cad9..e7405bd 100644
--- a/site/_docs/stack/scaling.md
+++ b/site/_docs/stack/scaling.md
@@ -28,7 +28,7 @@ It's important to note that the burden of scale isn't placed
wholly on your Apac
When thinking about how to scale your Apache Flagon Elastic stack, it's
important to note that Elasticsearch isn't a database, its a datastore, which
stores documents. Elastic is built on top of
[Lucene](http://lucene.apache.org/). That means that Apache Flagon "logs"
aren't logs once they're indexed in Elastic, they become searchable documents.
That's a huge strength (and why we chose Elastic), but it also means that
assumptions about resource consumptions based purely on records and fi [...]
-1. Given that documents are the atomic unit of storage in Elastic, *document
generation rate* is the most important consideration to scaling. Default
[Apache UserALE.js parameters]({{ '/docs/useralejs' | prepend: site.baseurl }})
produce a lot of [data]({{ '/docs/useralejs/dataschema' | prepend: site.baseurl
}}), even from single users in Apache UserALE.js. In fact, we used say that
"drinking from the fire-hose" didn't quite do our data-rate justice--we used to
say that opening up UserAL [...]
+1. Given that documents are the atomic unit of storage in Elastic, *document
generation rate* is the most important consideration to scaling. Default
[Apache UserALE.js parameters]({{ '/docs/useralejs' | prepend: site.baseurl }})
produce a lot of [data]({{ '/docs/useralejs/dataschema' | prepend: site.baseurl
}}), even from single users in Apache UserALE.js. In fact, we used say that
"drinking from the fire-hose" didn't quite do our data-rate justice--we used to
say that opening up UserAL [...]
![alt text][logBreakdown]
@@ -51,7 +51,7 @@ To generate log data, we'll be using our [UserALE.js
Example](https://github.com
Important: as noted in the instructions, you'll need to have collected
some log data to establish the index.
-1. **Once you've started up the ELK stack, take a look at the `userale` index
stats.**
+1. **Once you've started up Elasticsearch, Logstash, Kibana, and metricbead,
and confirm that you can see logs in Kibana, take a look at the `userale` index
stats.**
```shell
#Index Stats using Elastic's _stats API
$ curl localhost:9200/index_name/_stats?pretty=true
@@ -74,18 +74,169 @@ To generate log data, we'll be using our [UserALE.js
Example](https://github.com
"deleted" : 0
},
"store" : {
- "size_in_bytes" : 241212 #this is size of the index in bytes
(we've collect .24 MB).
+ "size_in_bytes" : 241212 #this is size of the index in bytes
(.24 MB).
...
```
+ Let's call this value a baseline for your benchmarking.
+
As you continue your benchmarking, the `userale` index "size_in_bytes"
will be one of your key metrics.
-1.
+
+1. **Next, let's see how much data UserALE.js produces with parameters set to
default on your page or app.** We'll call this: "drinking from the
flame-thrower".
+ Drop in a UserALE.js script-tag into your project (see [instructions]({{
'/docs/useralejs' | prepend: site.baseurl }})).
+
+ Here is an example of the script tag we're using in the UserALE.js Example
page for this test
+ ```
+ <script
+ src="file:/// ... /UserALEtest/userale-1.1.0.min.js"
+ data-url="http://localhost:8100/";
+ data-user="example-user"
+ data-version="1.1.1"
+ data-tool="Apache UserALE.js Example"
+ ></script>
+ ```
+ To get a conservative upper-bound, generate as many behaviors in your
page/app as you can (go nuts with mouse-overs, clicks and scrolls).
+
+ We did just that for 5 minutes solid. Here's what our `userale` index
looks now (#note annotations):
+
+ ```shell
+ "indices" : {
+ "userale" : {
+ "uuid" : "0h0Wxe2cSwqMALs4QCJ8Tw",
+ "primaries" : {
+ "docs" : {
+ "count" : 3282, #new userale document count
+ "deleted" : 0
+ },
+ "store" : {
+ "size_in_bytes" : 820978 #new size of the index (.82 MB)
+ ```
+ **Some simple math gives +1,998 documents (~2000) and +579,866 bytes (~.58
MB) generated with UserALE.js by one user in 5 mins**. Continuing our
ultra-conservative benchmarking, lets assume this data generation rate over an
8 hour period each day, which gives **~56 MB per day**. At 20 working days in
an average month, that gives **1.1 GB per month**.
+
+ Again, **this is ultra-conservative**; no one uses pages or applications
this way (even games). That's a lot of data for a single-user, but these are
valuable, worst-case-scenario upper-bounds to start with.
+
+ If you're a scientist or researcher, these figures might be fine,
especially if you want to pour over this data. If you're just interested in how
people are moving through your content, this is probably overkill. The biggest
culprit: take a look at our `Apache Flagon Page Usage Dashboard` to see.
+
+ ![alt text][mouseOverBench1]
+
+ Mouseovers accounted for a lot of the data we just produced. Raw mouseover
data is written to log very frequently and generates a lot of documents in
Elastic.
-[Elastic's Stats
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html)
-[Overview of Elastic's
APIs](https://www.datadoghq.com/blog/collect-elasticsearch-metrics/#index-stats-api)
+1. **Now, let's take the simplest approach to scaling back UserALE.js and see
how this changes your data generation rate**.
+
+ What we're doing is using the UserALE.js `configs` that can be passed
through the script tag to effectively "downsample" certain channels (e.g.,
mouseovers) that generate a lot of documents quickly.
+
+ Here's what our script tag looks like now:
+ ```
+ <script
+ src="file:/// ... /UserALEtest/userale-1.1.0.min.js"
+ data-url="http://localhost:8100/";
+ data-user="example-user"
+ data-version="1.1.1"
+ data-resolution=1000 #We've increased the delay between collection of
"high-resolution" events (e.g., mouseover, scrolls) by 100%.
+ data-tool="Apache UserALE.js Example"
+ ></script>
+ ```
+ Next, replicate your benchmarking, behaving in a similar way for the same
amount of time.
+
+ Here's what our `userale` index looks like now after another 5 minutes of
vigorous behavior.
+
+ ```shell
+ "indices" : {
+ "userale" : {
+ "uuid" : "0h0Wxe2cSwqMALs4QCJ8Tw",
+ "primaries" : {
+ "docs" : {
+ "count" : 4800, #new userale document count
+ "deleted" : 0
+ },
+ "store" : {
+ "size_in_bytes" : 115978 #new size of the index (1.1 MB)
+ ```
+ **Maths give us +1518 documents (~500 fewer than our first benchmark) and
+295,000 bytes (~.30 MB; ~40% less growth in the store size than previous)
generated by one user in 5 minutes.** That would be **28.8 MB per day** or
**~576 MB per working month**, per user. We've cut down our data generation
considerably by modifying one parameter in our script tag.
+
+ Why? look at the new proportion of mouseover events... we shaved those
down by ~50%. As that event stream has the highest data generation, net-net we
generated about 25% fewer documents with the same amount and duration of
behavior:
+
+ ![alt text][mouseOverBench2]
+
+1. **Still too much data? Below are some other things that you can do to curb
the growth of your `userale` index** as you continue benchmarking with your
page/app.
+
+ * If you don't need all the different types of events UserALE.js gathers
by default, you can easily curate the list of events you by modifying
[UserALE.js
source](https://github.com/apache/incubator-flagon-useralejs/tree/master/src)
(`attachHandlers.js`), then building a custom UserALE.js script.
+ * If you don't need both raw logs (specific records of events) and
interval logs (aggregates of specific events over a time interval), you can
easily drop intervals by modifying UserALE.js [UserALE.js
source](https://github.com/apache/incubator-flagon-useralejs/tree/master/src)
(`attachHandlers.js` or `packageLogs.js`), then building a custom UserALE.js
script.
+ * You can reduce the number of meta-data fields written to each UserALE.js
log, by modifying [UserALE.js
source](https://github.com/apache/incubator-flagon-useralejs/tree/master/src)
(`packageLogs.js`), then building a custom UserALE.js script.
+ * If you want different levels of logging granularity (more or less data)
for different pages within your site/app, you can build different versions of
UserALE.js to service different pages. Just name each version differently and
point to the one you want with script-tags, page-by-page.
+ * You can use our [API]({{ '/docs/useralejs/API' | prepend: site.baseurl
}}) for surgical precision in how specific elements (targets) on your page
generate data.
+
+### Other Tools to Support Benchmarking for Scaling
+1. In our benchmarking guide, we primarily used [Elastic's `Stats`
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html).
This provides very verbose output, which is pretty useful for understanding
all of your indexes, indexing behavior, and how data is being distributed
across nodes. You can use other [Elastic
APIs](https://www.datadoghq.com/blog/collect-elasticsearch-metrics/#index-stats-api)
for different views of what's going on inside your Apache F [...]
+
+ http://localhost:9200/_cat/indices?format=json&bytes=b&pretty
+
+ Output is very simple and index sizes stack on top of one another
+ ```shell
+ [
+ {
+ "health" : "green",
+ "status" : "open",
+ "index" : ".kibana_1",
+ "uuid" : "FnI_6AQYQEWp2mIxSlM8HQ",
+ "pri" : "1",
+ "rep" : "0",
+ "docs.count" : "184",
+ "docs.deleted" : "13",
+ "store.size" : "366457",
+ "pri.store.size" : "366457"
+ },
+ {
+ "health" : "yellow",
+ "status" : "open",
+ "index" : "metricbeat-6.6.2-2019.04.22",
+ "uuid" : "pDYNmzsxTFu9Z0Tc1_GdLw",
+ "pri" : "5",
+ "rep" : "1",
+ "docs.count" : "4687",
+ "docs.deleted" : "0",
+ "store.size" : "6309920",
+ "pri.store.size" : "6309920"
+ },
+ {
+ "health" : "green",
+ "status" : "open",
+ "index" : "userale",
+ "uuid" : "0h0Wxe2cSwqMALs4QCJ8Tw",
+ "pri" : "1",
+ "rep" : "0",
+ "docs.count" : "14018",
+ "docs.deleted" : "0",
+ "store.size" : "2235963",
+ "pri.store.size" : "2235963"
+ },
+ {
+ "health" : "yellow",
+ "status" : "open",
+ "index" : "metricbeat-6.6.2-2019.04.27",
+ "uuid" : "wTyUBXvNRMOwpR4lDF9BNA",
+ "pri" : "5",
+ "rep" : "1",
+ "docs.count" : "107124",
+ "docs.deleted" : "0",
+ "store.size" : "63263644",
+ "pri.store.size" : "63263644"
+ }
+ ]
+ ```
+1. There are a lot of other things going on inside your Apache Flagon Elastic
stack. Note above that depending on what you deploy (e.g., metricbeat) you may
have additional indices that need monitoring. For one, Metricbeat indices grow
fast and you'll want to benchmark that too. The Kibana index will grow as well,
but not as much; it depends on how how many additional objects you build (e.g.,
visualizations, indicies, dashboards). Both the `STATS` and `CAT` APIs are very
useful for seein [...]
+
+1. There are also a lot of things going on inside your server or Docker
containers, depending on how you deploy your Apache Flagon Elastic Stack. This
is why we've configured a metricbeat service to run with Flagon. In the same
way that you can benchmark your data generation rate while you generate
behavioral logs in your page/app, you can also look at how your Apache Flagon
stack is utilizing disk and compute resources. When you spin up our single-node
Apache Flagon container, make sure [...]
+
+ ![alt text][metricBeat]
+
+### Wonky Things that Can and Will Happen as You Benchmark
-[http://localhost:9200/userale/_stats?pretty=true](http://localhost:9200/userale/_stats?pretty=true)
+1. You just finished a benchmarking session after modifying UserALE.js to
produce less data. What you find is that your new store size is either
dramatically bigger than your last benchmark or smaller (which should be
impossible). What happened is a thing called
[merging](https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html).
It is a good thing, but it can be a confusing thing while you're benchmarking
against your store size. Essentially, each Elastic shard is [...]
+### Summary
+Benchmarking and adjusting your data-rate so that you can scale how you want
to is made very easy in Apache Flagon. We combine easily deployed and modified
capabilities with the power of Elastic's APIs and visualization capabilities.
Again, our single-node container is not a scaling solution. It's an ingredient,
a potent one, that not only serves as the building block for your cluster, but
also a benchmarking tool so that your Apache Flagon cluster meets your needs
for capability, scale [...]
+
+Subscribe to our [dev list]([email protected]) and
join the conversation!
-Subscribe to our [dev list]([email protected]) and
join the conversation!
\ No newline at end of file
diff --git a/site/_docs/useralejs/index.md b/site/_docs/useralejs/index.md
index 64e6afd..85dbfbc 100644
--- a/site/_docs/useralejs/index.md
+++ b/site/_docs/useralejs/index.md
@@ -10,7 +10,7 @@ priority: 0
*Note:* Work on UserALE.js' documentation is ongoing. Contributions are
welcome. To get involved, see our [Contributing]({{ '/docs/contributing' |
prepend: site.baseurl }}) guide.
### Include UserALE.js in your project
-To start logging with Apache UserALE.js, you can include our script directly
in your project. More details can be found in our
[README](https://github.com/apache/incubator-flagon-useralejs/blob/master/README.md).
+To start logging with Apache UserALE.js, you can include our script directly
in your project. More details can be found in our
[README](https://github.com/apache/incubator-flagon-useralejs/blob/master/README.md).
You can use our [build process]({{ '/docs/useralejs/build' | prepend:
site.baseurl }}) to create our script, or just use a sample in our
[repos](https://github.com/apache/incubator-flagon-useralejs/tree/master/build).
To include UserALE.js in a specific project, you'll need to deploy a version
of our minified UserALE.js script in an accessible location (e.g., webserver),
and simply include this script tag on the page:
