Fixed formatting and image
Project: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/commit/6b79095c Tree: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/tree/6b79095c Diff: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/diff/6b79095c Branch: refs/heads/master Commit: 6b79095cf6547da41bc55d7ea2cacc4e18899c17 Parents: d373226 Author: Chandni Singh <[email protected]> Authored: Wed Nov 4 18:53:00 2015 -0800 Committer: Thomas Weise <[email protected]> Committed: Sun Feb 28 22:46:36 2016 -0800 ---------------------------------------------------------------------- autometrics/autometrics.md | 52 ++++++++++++++++++++++++---------------- autometrics/images/adt.png | Bin 0 -> 169889 bytes 2 files changed, 31 insertions(+), 21 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/6b79095c/autometrics/autometrics.md ---------------------------------------------------------------------- diff --git a/autometrics/autometrics.md b/autometrics/autometrics.md index 8e7ffb2..364007f 100644 --- a/autometrics/autometrics.md +++ b/autometrics/autometrics.md @@ -51,12 +51,13 @@ For the `LineReceiver` operator, the application developer need not specify any ## Building custom aggregators -Platform cannot perform any meaningful aggregation for a non-numeric metric. In such case, the operator or application developer can write custom aggregators. Letâs say, in the +Platform cannot perform any meaningful aggregation for a non-numeric metric. In such case, the operator or application developer can write custom aggregators. Letâs say, if the `LineReceiver` was modified to -Retrieving AutoMetrics +# Retrieving AutoMetrics The Gateway REST API provides a way to retrieve the latest AutoMetrics for each logical operator. For example: +``` GET /ws/v2/applications/{appid}/logicalPlan/operators/{opName} { ... @@ -65,20 +66,23 @@ GET /ws/v2/applications/{appid}/logicalPlan/operators/{opName} } ... } +``` -System Metrics +# System Metrics System metrics are standard operator metrics provided by the system. Examples include: -processed tuples per second -emitted tuples per second -total tuples processed -total tuples emitted -latency -CPU percentage -failure count -checkpoint elapsed time +- processed tuples per second +- emitted tuples per second +- total tuples processed +- total tuples emitted +- latency +- CPU percentage +- failure count +- checkpoint elapsed time The Gateway REST API provides a way to retrieve the latest values for all of the above for each of the logical operators in the application. + +``` GET /ws/v2/applications/{appid}/logicalPlan/operators/{opName} { ... @@ -91,17 +95,18 @@ GET /ws/v2/applications/{appid}/logicalPlan/operators/{opName} "tuplesProcessedPSMA": "{tuplesProcessedPSMA}", ... } +``` However, just like AutoMetrics, the Gateway only provides the latest metrics. For historical metrics, we will need the help of App Data Tracker. -App Data Tracker +# App Data Tracker As discussed above, STRAM aggregates the AutoMetrics from physical operators (partitions) to something that makes sense in one logical operator. It pushes the aggregated AutoMetrics values using Websocket to the Gateway at every second along with system metrics for each operator. Gateway relays the information to an application called App Data Tracker. It is another Apex application that runs in the background and further aggregates the incoming values by time bucket and stores the values in HDHT. It also allows the outside to retrieve the aggregated AutoMetrics and system metrics through websocket interface. - - + App Data Tracker is enabled by having these properties in dt-site.xml: +```xml <property> <name>dt.appDataTracker.enable</name> <value>true</value> @@ -114,19 +119,21 @@ App Data Tracker is enabled by having these properties in dt-site.xml: <name>dt.attr.METRICS_TRANSPORT</name> <value>builtin:AppDataTrackerFeed</value> </property> +``` All the applications launched after the App Data Tracker is enabled will have metrics sent to it. -Note: The App Data Tracker will be shown running in dtManage as a âsystem appâ. It will show up if the âshow system appsâ button is pressed. +**Note**: The App Data Tracker will be shown running in dtManage as a âsystem appâ. It will show up if the âshow system appsâ button is pressed. + +By default, the time buckets App Data Tracker aggregates upon are one minute, one hour and one day. It can be overridden by changing the operator attribute `METRICS_DIMENSIONS_SCHEME`. -By default, the time buckets App Data Tracker aggregates upon are one minute, one hour and one day. It can be overridden by changing the operator attribute METRICS_DIMENSIONS_SCHEME. +Also by default, the app data tracker performs all these aggregations: SUM, MIN, MAX, AVG, COUNT, FIRST, LAST on all number metrics. You can also override by changing the same operator attribute `METRICS_DIMENSIONS_SCHEME`, provided the custom aggregator is known to the App Data Tracker. (See next section) -Also by default, the app data tracker performs all these aggregations: SUM, MIN, MAX, AVG, COUNT, FIRST, LAST on all number metrics. You can also override by changing the same operator attribute METRICS_DIMENSIONS_SCHEME, provided the custom aggregator is known to the App Data Tracker. (See next section) -Custom Aggregator in App Data Tracker +# Custom Aggregator in App Data Tracker Custom aggregators allow you to do your own custom computation on statistics generated by any of your applications. In order to implement a Custom aggregator you have to do two things: -Combining new inputs with the current aggregation -Combining two aggregations together into one aggregation +1. Combining new inputs with the current aggregation +2. Combining two aggregations together into one aggregation Letâs consider the case where we want to perform the following rolling average: @@ -134,6 +141,7 @@ Y_n = ½ * X_n + ½ * X_n-1 + ¼ * X_n-2 + â * X_n-3 +... This aggregation could be performed by the following Custom Aggregator: +```java @Name("IIRAVG") public class AggregatorIIRAVG extends AbstractIncrementalAggregator { @@ -163,7 +171,9 @@ public class AggregatorIIRAVG extends AbstractIncrementalAggregator aggregateHelper(destAgg, srcAgg); } } -Dashboards +``` + +# Dashboards With App Data Tracker enabled, you can visualize the AutoMetrics and system metrics in the Dashboards within dtManage. Refer back to the diagram in the App Data Tracker section, dtGateway relays queries and query results to and from the App Data Tracker. In this way, dtManage sends queries and receives results from the App Data Tracker via dtGateway and uses the results to let the user visualize the data. The Dashboards have some simple builtin widgets to visualize the data. Line charts and bar charts are some examples. http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/6b79095c/autometrics/images/adt.png ---------------------------------------------------------------------- diff --git a/autometrics/images/adt.png b/autometrics/images/adt.png new file mode 100644 index 0000000..5b0d8ea Binary files /dev/null and b/autometrics/images/adt.png differ
