Repository: eagle Updated Branches: refs/heads/master ee55054a7 -> 0c52711b8
[MINOR][DOC] Add oozie applicaton installation doc Author: chitin <[email protected]> Closes #904 from chitin/oozie-doc. Project: http://git-wip-us.apache.org/repos/asf/eagle/repo Commit: http://git-wip-us.apache.org/repos/asf/eagle/commit/0c52711b Tree: http://git-wip-us.apache.org/repos/asf/eagle/tree/0c52711b Diff: http://git-wip-us.apache.org/repos/asf/eagle/diff/0c52711b Branch: refs/heads/master Commit: 0c52711b897a4fd6d5aaf7330e89572582f1d189 Parents: ee55054 Author: chitin <[email protected]> Authored: Wed Apr 5 13:50:48 2017 +0800 Committer: Hao Chen <[email protected]> Committed: Wed Apr 5 13:50:48 2017 +0800 ---------------------------------------------------------------------- docs/docs/applications.md | 63 ++++++++++++++++++++++ docs/docs/include/images/oozie_audit_log.png | Bin 0 -> 41562 bytes docs/docs/include/images/oozie_install_1.png | Bin 0 -> 165098 bytes docs/docs/include/images/oozie_install_2.png | Bin 0 -> 153179 bytes docs/docs/include/images/oozie_install_3.png | Bin 0 -> 94200 bytes docs/docs/include/images/oozie_policy_1.png | Bin 0 -> 101463 bytes 6 files changed, 63 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/applications.md ---------------------------------------------------------------------- diff --git a/docs/docs/applications.md b/docs/docs/applications.md index 74efcc6..a54c236 100644 --- a/docs/docs/applications.md +++ b/docs/docs/applications.md @@ -70,6 +70,69 @@ The example policy monitors the 'delete' operation on hosts in 'SECURITY' zone. ## Questions on this application --- +# OOZIE Data Activity Monitoring + +## Monitor Requirements + +This application aims to monitor user activities on OOZIE via the oozie audit log. Once any abnormal user activity is detected, an alert is sent in several seconds. The whole pipeline of this application is + +* Kafka ingest: this application consumes data from Kafka. In other words, users have to stream the log into Kafka first. + +* Data re-procesing, which includes raw log parser, sensitivity information joiner. + +* Kafka sink: parsed data will flows into Kafka again, which will be consumed by the alert engine. + +* Policy evaluation: the alert engine (hosted in Alert Engine app) evaluates each data event to check if the data violate the user defined policy. An alert is generated if the data matches the policy. + + + + +## Setup & Installation + +* Choose a site to install this application. For example 'sandbox' + +* Install "Oozie Audit Log Monitor" app step by step + +  + +  + +  + + +## How to collect the log + +To collect the raw audit log on namenode servers, a log collector is needed. Users can choose any tools they like. There are some common solutions available: [logstash](https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html), [filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html), log4j appender, etcs. + +For detailed instruction, refer to: [How to stream audit log into Kafka](using-eagle/#how-to-stream-audit-log-into-kafka) + +## Sample policies + +### 1. monitor file/folder operations + +Delete a file/folder on HDFS. + +``` +from OOZIE_AUDIT_LOG_STREAM_SANDBOX[ip=='localhost'] select * insert into oozie_audit_log_enriched_stream_out +``` + +OOZIE_AUDIT_LOG_STREAM_SANDBOX is the input stream name, and oozie_audit_log_enriched_stream_out is the output stream name, the content between [] is the monitoring conditions. `user`, `ip`, `jobId`, `operation` and `timestamp` is the fields of oozie audit logs. + +  + +### 2. classify the file/folder on OOZIE + +Users may want to mark some jobId on OOZIE as sensitive job. For example, by marking '0000101-161115152703493-oozie-oozi-C' as "BASE", users can monitor any operations they care about on these jobId. + +``` +from OOZIE_AUDIT_LOG_STREAM_SANDBOX[sensitivityType=='BASE' and cmd=='kill')] select * group by user insert into oozie_audit_log_enriched_stream_out +``` +The example policy monitors the 'kill' operation on jobId under "BASE". + +# Questions on this application + +--- + # JMX Monitoring http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/include/images/oozie_audit_log.png ---------------------------------------------------------------------- diff --git a/docs/docs/include/images/oozie_audit_log.png b/docs/docs/include/images/oozie_audit_log.png new file mode 100644 index 0000000..42d38f6 Binary files /dev/null and b/docs/docs/include/images/oozie_audit_log.png differ http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/include/images/oozie_install_1.png ---------------------------------------------------------------------- diff --git a/docs/docs/include/images/oozie_install_1.png b/docs/docs/include/images/oozie_install_1.png new file mode 100644 index 0000000..238f1f5 Binary files /dev/null and b/docs/docs/include/images/oozie_install_1.png differ http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/include/images/oozie_install_2.png ---------------------------------------------------------------------- diff --git a/docs/docs/include/images/oozie_install_2.png b/docs/docs/include/images/oozie_install_2.png new file mode 100644 index 0000000..da4c110 Binary files /dev/null and b/docs/docs/include/images/oozie_install_2.png differ http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/include/images/oozie_install_3.png ---------------------------------------------------------------------- diff --git a/docs/docs/include/images/oozie_install_3.png b/docs/docs/include/images/oozie_install_3.png new file mode 100644 index 0000000..c59219e Binary files /dev/null and b/docs/docs/include/images/oozie_install_3.png differ http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/include/images/oozie_policy_1.png ---------------------------------------------------------------------- diff --git a/docs/docs/include/images/oozie_policy_1.png b/docs/docs/include/images/oozie_policy_1.png new file mode 100644 index 0000000..e75dfb8 Binary files /dev/null and b/docs/docs/include/images/oozie_policy_1.png differ
