Repository: eagle Updated Branches: refs/heads/master e2fbb8613 -> e963abeda
[MINOR] update eagle-assembly/src/main/doc/README.md * Update eagle-assembly/src/main/doc/README.md * Fix a bug in MRHistoryJobApplicationProvider.xml Author: Zhao, Qingwen <[email protected]> Closes #946 from qingwen220/minor. Project: http://git-wip-us.apache.org/repos/asf/eagle/repo Commit: http://git-wip-us.apache.org/repos/asf/eagle/commit/e963abed Tree: http://git-wip-us.apache.org/repos/asf/eagle/tree/e963abed Diff: http://git-wip-us.apache.org/repos/asf/eagle/diff/e963abed Branch: refs/heads/master Commit: e963abeda31db0f561b401cebe0709a551afdd62 Parents: e2fbb86 Author: Zhao, Qingwen <[email protected]> Authored: Thu Jun 8 21:10:40 2017 +0800 Committer: Zhao, Qingwen <[email protected]> Committed: Thu Jun 8 21:10:40 2017 +0800 ---------------------------------------------------------------------- eagle-assembly/src/main/doc/README.md | 271 +++++++++++++++---- ....history.MRHistoryJobApplicationProvider.xml | 4 +- 2 files changed, 213 insertions(+), 62 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/eagle/blob/e963abed/eagle-assembly/src/main/doc/README.md ---------------------------------------------------------------------- diff --git a/eagle-assembly/src/main/doc/README.md b/eagle-assembly/src/main/doc/README.md index d9ee25f..21950a9 100644 --- a/eagle-assembly/src/main/doc/README.md +++ b/eagle-assembly/src/main/doc/README.md @@ -1,99 +1,250 @@ <!-- -{% comment %} -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -{% endcomment %} +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to you under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. --> -Eagle User Guide -======================== +# Architecture -Prerequisites -------------- + + +### Eagle Apps + +* Security * Hadoop -* HBase -* Storm -* Spark -* Kafka +* Operational Intelligence -Eagle requires you to have access on Hadoop CLI, where you have full permissions to HDFS, Storm, HBase and Kafka. To make things easier, we strongly recommend you to start Eagle on a hadoop sandbox such as http://hortonworks.com/products/hortonworks-sandbox/ +For more applications, see [Applications](applications). +### Eagle Interface -Build ------ +* REST Service +* Management UI +* Customizable Analytics Visualization -* Download the latest version of Eagle source code. +### Eagle Integration - git clone [email protected]:eagle/eagle.git +* [Apache Ambari](https://ambari.apache.org) +* [Docker](https://www.docker.com) +* [Apache Ranger](http://ranger.apache.org) +* [Dataguise](https://www.dataguise.com) +### Eagle Framework -* Build the source code, and a tar.gz package will be generated under eagle-assembly/target. +Eagle has multiple distributed real-time frameworks for efficiently developing highly scalable monitoring applications. + +#### Alert Engine - mvn clean compile install -DskipTests + -Installation ------------ -* Copy this package onto the sandbox. +* Real-time: Apache Storm (Execution Engine) + Kafka (Message Bus) +* Declarative Policy: SQL (CEP) on Streaming + from hadoopJmxMetricEventStream + [metric == "hadoop.namenode.fsnamesystemstate.capacityused" and value > 0.9] + select metric, host, value, timestamp, component, site + insert into alertStream; - scp -P 2222 eagle/eagle-assembly/target/eagle-0.1.0-bin.tar.gz [email protected]:/usr/hdp/current/. +* Dynamical onboarding & correlation +* No downtime migration and upgrading -* Run Eagle patch installation at the first time, and restart HDFS namenode. +#### Storage Engine - bin/eagle-patch-install.sh + -* Start Storm, HBase, and Kafka via Ambari Web UI. Make sure the user has the privilege to run Storm, HBase, and Kafka cmd in shell, and with full permissions to access HBase, such as creating tables. Check the installation & running status of the required services. +* Light-weight ORM Framework for HBase/RDMBS + + @Table("HbaseTableName") + @ColumnFamily("ColumnFamily") + @Prefix("RowkeyPrefix") + @Service("UniqueEntitytServiceName") + @JsonIgnoreProperties(ignoreUnknown = true) + @TimeSeries(false) + @Indexes({ + @Index(name="Index_1_alertExecutorId", columns = { "alertExecutorID" }, unique = true)}) + public class AlertDefinitionAPIEntity extends TaggedLogAPIEntity{ + @Column("a") + private String desc; - bin/eagle-check-env.sh +* Full-function SQL-Like REST Query + Query=UniqueEntitytServiceName[@site="sandbox"]{*} -* Create necessary HBase tables for Eagle. +* Optimized Rowkey design for time-series data, optimized for metric/entity/log, etc. different storage types + + Rowkey ::= Prefix | Partition Keys | timestamp | tagName | tagValue | ⦠+ - bin/eagle-service-init.sh +* Secondary Index Support + @Indexes({@Index(name="INDEX_NAME", columns = { "SECONDARY_INDEX_COLUMN_NAME" }, unique = true/false)}) + +* Native HBase Coprocessor + org.apache.eagle.storage.hbase.query.coprocessor.AggregateProtocolEndPoint -* Start Eagle service. +#### UI Framework - bin/eagle-service.sh start - +Eagle UI is consist of following parts: -* Create Kafka topics and topology metadata for Eagle. +* Eagle Main UI +* Eagle App Portal/Dashboard/Widgets +* Eagle Customized Dashboard - bin/eagle-topology-init.sh +#### Application Framework +##### Application -* Start Eagle topology, which will submit the topology to Storm via the Storm CLI tools. You can check it with storm UI. +An "Application" or "App" is composed of data integration, policies and insights for one data source. - bin/eagle-topology.sh [--jar <jarName>] [--main <mainClass>] [--topology <topologyName>] start +##### Application Descriptor +An "Application Descriptor" is a static packaged metadata information consist of basic information like type, name, version, description, and application process, configuration, streams, docs, policies and so on. -Now you can let Eagle to monitor by creating your own policy! +Here is an example ApplicationDesc of `JPM_WEB_APP` + { + type: "JPM_WEB_APP", + name: "Job Performance Monitoring Web ", + version: "0.5.0-incubating", + description: null, + appClass: "org.apache.eagle.app.StaticApplication", + jarPath: "/opt/eagle/0.5.0-incubating-SNAPSHOT-build-20161103T0332/eagle-0.5.0-incubating-SNAPSHOT/lib/eagle-topology-0.5.0-incubating-SNAPSHOT-hadoop-2.4.1-11-assembly.jar", + viewPath: "/apps/jpm", + providerClass: "org.apache.eagle.app.jpm.JPMWebApplicationProvider", + configuration: { + properties: [{ + name: "service.host", + displayName: "Eagle Service Host", + value: "localhost", + description: "Eagle Service Host, default: localhost", + required: false + }, { + name: "service.port", + displayName: "Eagle Service Port", + value: "8080", + description: "Eagle Service Port, default: 8080", + required: false + }] + }, + streams: null, + docs: null, + executable: false, + dependencies: [{ + type: "MR_RUNNING_JOB_APP", + version: "0.5.0-incubating", + required: true + }, { + type: "MR_HISTORY_JOB_APP", + version: "0.5.0-incubating", + required: true + }] + } + -Sandbox Starter ---------------- +##### Application Provider -* startup Eagle service & topology +Appilcation Provider is a package management and loading mechanism leveraging [Java SPI](https://docs.oracle.com/javase/tutorial/ext/basics/spi.html). + +For example, in file `META-INF/services/org.apache.eagle.app.spi.ApplicationProvider`, place the full class name of an application provider: - examples/eagle-sandbox-starter.sh - -* check eagle UI <http://127.0.0.1:9099/eagle-service> + org.apache.eagle.app.jpm.JPMWebApplicationProvider + + +--- + +# Concepts + +* Here are some terms we are using in Apache Eagle (incubating, called Eagle in the following), please check them for your reference. They are basic knowledge of Eagle which also will help to well understand Eagle. + +## Site + +* A site can be considered as a physical data center. Big data platform e.g. Hadoop may be deployed to multiple data centers in an enterprise. + +## Application + +* An "Application" or "App" is composed of data integration, policies and insights for one data source. + +## Policy + +* A "Policy" defines the rule to alert. Policy can be simply a filter expression or a complex window based aggregation rules etc. + +## Alerts - * If you get a 404 Error when trying to access the UI, add port 9099 to "Settings->Network->Advanced->Port Forwarding" in VirtualBox. (See step 4 in "Setup Hadoop Environment" section in <https://eagle.incubator.apache.org/docs/quick-start.html>) +* An "Alert" is an real-time event detected with certain alert policy or correlation logic, with different severity levels like INFO/WARNING/DANGER. + +## Data Source + +* A "Data Source" is a monitoring target data. Eagle supports many data sources HDFS audit logs, Hive2 query, MapReduce job etc. + +## Stream + +* A "Stream" is the streaming data from a data source. Each data source has its own stream. + +--- + +# Quick Start + +## Deployment + +### Prerequisites + +Eagle requires the following dependencies: + +* For streaming platform dependencies + * Storm: 0.9.3 or later + * Hadoop: 2.6.x or later + * Hbase: 0.98.x or later + * Kafka: 0.8.x or later + * Zookeeper: 3.4.6 or later + * Java: 1.8.x +* For metadata database dependencies (Choose one of them) + * MangoDB 3.2.2 or later + * Installation is required + * Mysql 5.1.x or later + * Installation is required + +Notice: +> Storm 0.9.x does NOT support JDK8. You can replace asm-4.0.jar with asm-all-5.0.jar in the storm lib directory. +> Then restart other services(nimbus/ui/supervisor). + + +### Installation + +##### Build Eagle + +* Download the latest version of Eagle source code. -* Take the following actions which will violate and obey the sample policy. - * Violation Action: hdfs dfs -ls unknown - * Violation Action: hdfs dfs -touchz /tmp/private - * Obey Action: hdfs dfs -cat /tmp/private + git clone https://github.com/apache/incubator-eagle.git + +* Build the source code, and a tar.gz package will be generated under eagle-server-assembly/target + + mvn clean install -DskipTests + +##### Deploy Eagle +* Copy binary package to your server machine. In the package, you should find: + * __bin/__: scripts used for start eagle server + * __conf/__: default configurations for eagle server setup. + * __lib/__ : all included software packages for eagle server +* Change configurations under `conf/` + * __eagle.conf__ + * __server.yml__ +* Run eagle-server.sh + + ./bin/eagle-server.sh start + +* Check eagle server + * Visit http://host:port/ in your web browser. + +## Setup Your Monitoring Case +`Placeholder for topic: Setup Your Monitoring Case` \ No newline at end of file http://git-wip-us.apache.org/repos/asf/eagle/blob/e963abed/eagle-jpm/eagle-jpm-mr-history/src/main/resources/META-INF/providers/org.apache.eagle.jpm.mr.history.MRHistoryJobApplicationProvider.xml ---------------------------------------------------------------------- diff --git a/eagle-jpm/eagle-jpm-mr-history/src/main/resources/META-INF/providers/org.apache.eagle.jpm.mr.history.MRHistoryJobApplicationProvider.xml b/eagle-jpm/eagle-jpm-mr-history/src/main/resources/META-INF/providers/org.apache.eagle.jpm.mr.history.MRHistoryJobApplicationProvider.xml index 90c002a..cff312b 100644 --- a/eagle-jpm/eagle-jpm-mr-history/src/main/resources/META-INF/providers/org.apache.eagle.jpm.mr.history.MRHistoryJobApplicationProvider.xml +++ b/eagle-jpm/eagle-jpm-mr-history/src/main/resources/META-INF/providers/org.apache.eagle.jpm.mr.history.MRHistoryJobApplicationProvider.xml @@ -216,12 +216,12 @@ <defaultValue>0.0</defaultValue> </column> <column> - <name>avgOpsPerMapTask</name> + <name>avgOpsPerMap</name> <type>double</type> <defaultValue>0.0</defaultValue> </column> <column> - <name>avgOpsPerReduceTask</name> + <name>avgOpsPerReduce</name> <type>double</type> <defaultValue>0.0</defaultValue> </column>
