[EAGLE-279] Create Documentation for Sandbox Quick starter and JMX monitring

Create Documentation for Sandbox Quick starter and JMX monitring

Author: hdendukuri <hdenduk...@ebay.com>

Closes #159 from hdendukuri/master.


Project: http://git-wip-us.apache.org/repos/asf/incubator-eagle/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-eagle/commit/df576452
Tree: http://git-wip-us.apache.org/repos/asf/incubator-eagle/tree/df576452
Diff: http://git-wip-us.apache.org/repos/asf/incubator-eagle/diff/df576452

Branch: refs/heads/master
Commit: df576452f174166407dfaa7a705b7f263f932f33
Parents: 9849303
Author: hdendukuri <hdenduk...@ebay.com>
Authored: Fri May 27 19:36:26 2016 +0800
Committer: Hao Chen <h...@apache.org>
Committed: Fri May 27 19:36:26 2016 +0800

----------------------------------------------------------------------
 eagle-docs/images/eagle-service.png           | Bin 0 -> 449283 bytes
 eagle-docs/images/hbase-superuser.png         | Bin 0 -> 147292 bytes
 eagle-docs/images/hdfs-env-conf.png           | Bin 0 -> 460829 bytes
 eagle-docs/images/hdfs-env-conf2.png          | Bin 0 -> 265131 bytes
 eagle-docs/images/hdfs-log4j-conf.png         | Bin 0 -> 359551 bytes
 eagle-docs/images/new-site.png                | Bin 0 -> 61032 bytes
 eagle-docs/images/nn-restart.png              | Bin 0 -> 79477 bytes
 eagle-docs/images/start-storm.png             | Bin 0 -> 500714 bytes
 eagle-docs/userguide/jmx-metric-monitoring.md |  77 ++++++++++++
 eagle-docs/userguide/sandbox-quick-start.md   | 140 +++++++++++++++++++++
 10 files changed, 217 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/eagle-service.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/eagle-service.png 
b/eagle-docs/images/eagle-service.png
new file mode 100644
index 0000000..c649e36
Binary files /dev/null and b/eagle-docs/images/eagle-service.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/hbase-superuser.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hbase-superuser.png 
b/eagle-docs/images/hbase-superuser.png
new file mode 100644
index 0000000..71b4a04
Binary files /dev/null and b/eagle-docs/images/hbase-superuser.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/hdfs-env-conf.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hdfs-env-conf.png 
b/eagle-docs/images/hdfs-env-conf.png
new file mode 100644
index 0000000..b1ba80e
Binary files /dev/null and b/eagle-docs/images/hdfs-env-conf.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/hdfs-env-conf2.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hdfs-env-conf2.png 
b/eagle-docs/images/hdfs-env-conf2.png
new file mode 100644
index 0000000..069acee
Binary files /dev/null and b/eagle-docs/images/hdfs-env-conf2.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/hdfs-log4j-conf.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hdfs-log4j-conf.png 
b/eagle-docs/images/hdfs-log4j-conf.png
new file mode 100644
index 0000000..0eade68
Binary files /dev/null and b/eagle-docs/images/hdfs-log4j-conf.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/new-site.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/new-site.png b/eagle-docs/images/new-site.png
new file mode 100644
index 0000000..793aa18
Binary files /dev/null and b/eagle-docs/images/new-site.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/nn-restart.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/nn-restart.png b/eagle-docs/images/nn-restart.png
new file mode 100644
index 0000000..562641f
Binary files /dev/null and b/eagle-docs/images/nn-restart.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/start-storm.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/start-storm.png 
b/eagle-docs/images/start-storm.png
new file mode 100644
index 0000000..705b60e
Binary files /dev/null and b/eagle-docs/images/start-storm.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/userguide/jmx-metric-monitoring.md
----------------------------------------------------------------------
diff --git a/eagle-docs/userguide/jmx-metric-monitoring.md 
b/eagle-docs/userguide/jmx-metric-monitoring.md
new file mode 100644
index 0000000..1a31d63
--- /dev/null
+++ b/eagle-docs/userguide/jmx-metric-monitoring.md
@@ -0,0 +1,77 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+
+
+---
+layout: doc
+title:  "JMX Metric Monitoring" 
+permalink: /docs/jmx-metric-monitoring.html
+---
+-->
+
+JMX metric for hadoop namenode url 
[http://127.0.0.1:50070/jmx](http://127.0.0.1:50070/jmx) can be monitored using 
Eagle. Follow below steps to enable this feature in Eagle.    
+
+1. Install Python script (To populate JMX metric values to Kafka message topic 
periodically.)
+2. Deploy "hadoopjmx" storm topology.
+3. Create new site and policy in UI
+4. Validate policy alert.
+
+<br/>
+
+
+### **Prerequisite**
+* Complete the setup from 
[sandbox-quick-start.md](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/userguide/sandbox-quick-start.md)
      
+
+<br/>
+
+
+### **Setup**
+From Hortonworks sandbox just run below setup script to Install Pyton JMX 
script, Create Kafka topic, update Hbase tables and deploy "hadoopjmx" storm 
topology. 
+
+    $ /usr/hdp/current/eagle/examples/hadoop-metric-sandbox-starter.sh
+    $ /usr/hdp/current/eagle/examples/hadoop-metric-policy-create.sh
+
+<br/>
+
+
+### **Application Setup in UI**
+1. Login to Eagle UI 
[http://localhost:9099/eagle-service/](http://localhost:9099/eagle-service/) 
username and password as "admin" and "secret"
+2. Click on "Admin" from top right and click "Management" button.
+3. On Admin management page add "New Site" name "hadoopJmxMetricDataSource" by 
clicking on "New Site" link.
+![add 
superuser](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/new-site.png)
+4. Save the changes.
+5. On eagle home page you should see new tab called "METRIC", besides "DAM".
+6. Click on "JmxMetricMonitor" under "METRIC".
+ 
+You should see safeModePolicy policy.  
+
+<br/>
+
+
+### **Demo** 
+
+* First make sure that Kafka topic "nn_jmx_metric_sandbox" is populated with 
JMX metric data periodically.(To make sure that python script is running)
+ 
+        $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper 
sandbox.hortonworks.com:2181 --topic nn_jmx_metric_sandbox
+
+* Genrate Alert by producing alert triggering message in Kafka topic.  
+
+
+        $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-producer.sh --broker-list 
sandbox.hortonworks.com:6667 --topic nn_jmx_metric_sandbox
+        $ {"host": "localhost", "timestamp": 1457033916718, "metric": 
"hadoop.namenode.fsnamesystemstate.fsstate", "component": "namenode", "site": 
"sandbox", "value": 1.0}
+  

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/userguide/sandbox-quick-start.md
----------------------------------------------------------------------
diff --git a/eagle-docs/userguide/sandbox-quick-start.md 
b/eagle-docs/userguide/sandbox-quick-start.md
new file mode 100644
index 0000000..8ad41d3
--- /dev/null
+++ b/eagle-docs/userguide/sandbox-quick-start.md
@@ -0,0 +1,140 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+Guide To Install Eagle Hortonworks sandbox. 
+===============================================
+* Prerequisite
+* Download + Patch + Build
+* Setup Hadoop Environment.
+* Stream HDFS audit logs into Kafka.
+* Install Eagle.
+* Demos "HDFS File System" & "Hive" monitoring
+
+### **Prerequisite**
+* To install Eagle on a sandbox you need to run a HDP sandbox image in a 
virtual machine with 8GB memory recommended.
+       1. Get Virtual Box or VMware [Virtualization 
environment](http://hortonworks.com/products/hortonworks-sandbox/#install)
+       2. Get [Hortonworks Sandbox v 
2.2.4](http://hortonworks.com/products/hortonworks-sandbox/#archive)
+* JDK 1.7  
+* NPM (On MAC OS try "brew install node")      
+<br/>
+
+### **Download + Patch + Build**
+* Download latest Eagle source released From Apache 
[[Tar]](http://www-us.apache.org/dist/incubator/eagle/apache-eagle-0.3.0-incubating/apache-eagle-0.3.0-incubating-src.tar.gz)
 , 
[[MD5]](http://www-us.apache.org/dist/incubator/eagle/apache-eagle-0.3.0-incubating/apache-eagle-0.3.0-incubating-src.tar.gz.md5)
 
+* Build manually with [Apache Maven](https://maven.apache.org/):
+
+     >$ tar -zxvf apache-eagle-0.3.0-incubating-src.tar.gz <br/>
+     >$ cd incubator-eagle-release-0.3.0-rc3 <br/>
+     >$ curl -O 
https://patch-diff.githubusercontent.com/raw/apache/incubator-eagle/pull/150.patch
 <br/>
+     >$ git apply 150.patch <br/>
+     >$ mvn clean package -DskipTests <br/>
+     
+* After building successfully, you will get the tarball under 
`eagle-assembly/target/` named as `eagle-0.3.0-incubating-bin.tar.gz`
+<br/>
+
+### **Setup Hadoop Environment In Sandbox**
+1. Launch Ambari to manage the Hadoop environment
+   * Enable Ambari in sandbox http://127.0.0.1:8000 (Click on Enable Button)
+   * Login to Ambari UI http://127.0.0.1:8080/ with username and password as 
"admin"
+2. Grant root as HBase superuser via Ambari
+![add 
superuser](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/hbase-superuser.png)
+3. Start HBase,Storm & Kafka from Ambari. Showing Storm as an example below. 
+![Restart 
Services](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/start-storm.png
 "Services")
+4. If the NAT network is used in a virtual machine, it's required to add port 
9099 to forwarding ports
+  ![Forwarding 
Port](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/eagle-service.png)
+
+<br/>
+
+### **Stream HDFS audit logs into Kafka**   
+ 
+**Note**: This section is only needed for "HDFS File System" Monitoring.
+ 
+* **Step 1**: Configure Advanced hadoop-log4j via <a 
href="http://localhost:8080/#/main/services/HDFS/configs"; 
target="_blank">Ambari UI</a>, and add below "KAFKA_HDFS_AUDIT" log4j appender 
to hdfs audit logging.
+
+       
log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
+       log4j.appender.KAFKA_HDFS_AUDIT.Topic=sandbox_hdfs_audit_log
+       log4j.appender.KAFKA_HDFS_AUDIT.BrokerList=sandbox.hortonworks.com:6667
+       
log4j.appender.KAFKA_HDFS_AUDIT.KeyClass=org.apache.eagle.log4j.kafka.hadoop.AuditLogKeyer
+       log4j.appender.KAFKA_HDFS_AUDIT.Layout=org.apache.log4j.PatternLayout
+       log4j.appender.KAFKA_HDFS_AUDIT.Layout.ConversionPattern=%d{ISO8601} %p 
%c{2}: %m%n
+       log4j.appender.KAFKA_HDFS_AUDIT.ProducerType=async
+
+    ![HDFS LOG4J 
Configuration](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/hdfs-log4j-conf.png
 "hdfslog4jconf")
+
+* **Step 2**: Edit Advanced hadoop-env via <a 
href="http://localhost:8080/#/main/services/HDFS/configs"; 
target="_blank">Ambari UI</a>, and add the reference to KAFKA_HDFS_AUDIT to 
HADOOP_NAMENODE_OPTS.
+
+      -Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT
+
+    ![HDFS Environment 
Configuration](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/hdfs-env-conf.png
 "hdfsenvconf")
+
+* **Step 3**: Edit Advanced hadoop-env via <a 
href="http://localhost:8080/#/main/services/HDFS/configs"; 
target="_blank">Ambari UI</a>, and append the following command to it.
+
+      export 
HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/eagle/lib/log4jkafka/lib/*
+
+    ![HDFS Environment 
Configuration](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/hdfs-env-conf2.png
 "hdfsenvconf2")
+
+* **Step 4**: save the changes 
+
+* **Step 5**: "Restart All" Storm & Kafka from Ambari. (Similar to starting 
server in pervious step except make sure to click on "Restart All")
+
+* **Step 6**: Restart name node 
+![Restart 
Services](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/nn-restart.png
 "Services")
+
+* **Step 7**: Check whether logs from are flowing into topic 
`sandbox_hdfs_audit_log`
+      
+      > /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper 
sandbox.hortonworks.com:2181 --topic sandbox_hdfs_audit_log      
+      
+<br/>
+
+### **Install Eagle**
+The following installation actually contains installing and setting up a 
sandbox site with HdfsAuditLog & HiveQueryLog  data sources. 
+    
+    >$ scp -P 2222  /eagle-assembly/target/eagle-0.3.0-incubating-bin.tar.gz 
root@127.0.0.1:/root/ <br/>
+    >$ ssh root@127.0.0.1 -p 2222 <br/>
+    >$ tar -zxvf eagle-0.3.0-incubating-bin.tar.gz <br/>
+    >$ mv eagle-0.3.0-incubating eagle <br/>
+    >$ mv eagle /usr/hdp/current/ <br/>
+    >$ cd /usr/hdp/current/eagle <br/>
+    >$ examples/eagle-sandbox-starter.sh <br/>
+
+<br/>
+
+### **Demos**
+* Login to Eagle UI 
[http://localhost:9099/eagle-service/](http://localhost:9099/eagle-service/) 
username and password as "admin" and "secret"
+* **HDFS**:
+       1. Click on menu "DAM" and select "HDFS" to view HDFS policy
+       2. You should see policy with name "viewPrivate". This Policy generates 
alert when any user reads HDFS file name "private" under "tmp" folder.
+       3. In sandbox read restricted HDFS file "/tmp/private" by using command 
+       
+          > hadoop fs -cat "/tmp/private"
+
+       From UI click on alert tab and you should see alert for the attempt to 
read restricted file.  
+* **Hive**:
+       1. Click on menu "DAM" and select "Hive" to view Hive policy
+       2. You should see policy with name "queryPhoneNumber". This Policy 
generates alert when hive table with sensitivity(Phone_Number) information is 
queried. 
+       3. In sandbox read restricted sensitive HIVE column.
+       
+       >$ su hive <br/>
+       >$ hive <br/>
+       >$ set hive.execution.engine=mr; <br/>
+       >$ use xademo; <br/>
+       >$ select a.phone_number from customer_details a, call_detail_records b 
where a.phone_number=b.phone_number; <br/>
+
+    From UI click on alert tab and you should see alert for your attempt to 
dfsf read restricted column.  
+
+<br/>

Reply via email to