Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The following page has been changed by JiaqiTan: http://wiki.apache.org/hadoop/Anomaly_Detection_Framework_with_Chukwa ------------------------------------------------------------------------------ == Implementation == - === Hadoop anomaly detection and behavioral visualization === + === Hadoop Anomaly Detection and Behavioral Visualization (Fingerpointing) === Current active developments for the Chukwa Anomaly Detection Framework are for detecting anomalies in Hadoop based on the following tools/concepts from the CMU [http://www.ece.cmu.edu/~fingerpointing/ Fingerpointing project]: @@ -45, +45 @@ This visualization shows the detailed task-level progress of MapReduce jobs across nodes in the cluster. 1. ({{{FSMBuilder}}} MapReduce job, available soon) SALSA is used to extract state-machine views of Hadoop's execution - uses post-Demux output; uses {{{JobData/JobHistory}}} - 1. ([http://issues.apache.org/jira/browse/CHUKWA-279 CHUKWA-279]) State-machine data from {{{FSMBuilder}}} is loaded into RDBMS using MDL + 1. State-machine data from {{{FSMBuilder}}} is loaded into RDBMS using MDL - 1. ([http://issues.apache.org/jira/browse/CHUKWA-279 CHUKWA-279]) Raw state-machine views visualized using Swimlanes visualization HICC widget which reads data from RDBMS + 1. Raw state-machine views visualized using Swimlanes visualization HICC widget which reads data from RDBMS ==== MIROS (N x N heatmaps) Visualization ==== @@ -65, +65 @@ 1. Load anomaly data into RDBMS using MDL 1. Visualization of heatmaps using HICC widget + == Usage == + + === Hadoop Anomaly Detection and Behavioral Visualization (Fingerpointing) === + + ==== Swimlanes Visualization ==== + + This visualization is generated from the output of the Demux operation. The steps (mostly envisioned to be automated) involved in generating the visualization are: + + 1. Generate state-machine views using {{{FSMBuilder}}} (Currently unavailable, pending feature additions to the PostProcessor to support non-MDL tasks): Read post-Demux data (SequenceFiles of ChukwaRecords of JobData data) as input, write state-machine view as SequenceFiles of ChukwaRecords of states. (Unsupported at time of writing, but will be available soon) + 1. Load states into database using MDL ([http://issues.apache.org/jira/browse/CHUKWA-279 CHUKWA-279]) + 1. Load Swimlanes widget in HICC ([http://issues.apache.org/jira/browse/CHUKWA-279 CHUKWA-279]) +
