> On Jan. 30, 2016, 6:28 a.m., Lenni Kuff wrote: > > This is awesome! Assume this works for all execution engines? Left some > > comments.
Forgot to answer the question.. yes it works across execution engine. Today the stage page is quite limited (name, start/end time, state, and other basic metadata). Further work is actually required to have execution-engine specific things in stage-tab, like YARN URL, number of mapper/reducer for MR, number of executor for Spark, or even Tez.. the difficulty is that most of these engines the job submission is in a separate process than HS2. - Szehon ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43008/#review117098 ----------------------------------------------------------- On Feb. 1, 2016, 8:05 p.m., Szehon Ho wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/43008/ > ----------------------------------------------------------- > > (Updated Feb. 1, 2016, 8:05 p.m.) > > > Review request for hive. > > > Bugs: HIVE-12952 > https://issues.apache.org/jira/browse/HIVE-12952 > > > Repository: hive-git > > > Description > ------- > > This patch shows a query sub-page on WebUI, with detailed information of > query on differnt tabs: > > 1. Tabl- Base Info, ie user, query string, query id, begin time, end time, > execution engine, error (if any) > 2. Tab2- Query Plan > 3. Tab3- Stages (MR jobs), their progress and info > 4. Tab4- Call trace info captured from HMSClient and PerfLogger. > > Implementation notes: > The UI design choices are inspired from Impala, and HBase. This, like HBase > webui, uses Jamon, which is a superset of JSP and makes dynamic content a lot > easier. As such, brought in jamon dependency and also js bootstrap libraries > to support the dynamic tabs. > > On Hive side, refactored webui query logic into following classes: > SQLoperationDisplay (info captured from SQLOperation), QueryDisplay (info > captured from Driver). > > > TODO: > 1. Hard to get more MR job information for the stages including a > job-tracking url, due to MR JobSubmission being a separate process, need to > think about it. Same for Spark/tez. > 2. The explain plan might be a bit bulky and consume a bit of memory (though > can tune with "hive.server2.webui.max.historic.queries"). Perhaps in future > we can spill to local disk, and stream from there. This might also help with > (1), if we don't want to implement inter-process communciations. > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ff376a8 > common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java d4194cf > common/src/java/org/apache/hive/http/HttpServer.java 9e23b11 > pom.xml 802d3d4 > ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 > ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 0bab769 > service/pom.xml b2e3a84 > service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon PRE-CREATION > service/src/java/org/apache/hive/service/cli/operation/Operation.java > 0c263cf > > service/src/java/org/apache/hive/service/cli/operation/OperationManager.java > f1ce6f6 > service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java > 01b1d3d > > service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplay.java > PRE-CREATION > > service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplayCache.java > PRE-CREATION > > service/src/java/org/apache/hive/service/cli/operation/SQLOperationInfo.java > 179f6dd > service/src/java/org/apache/hive/service/server/HiveServer2.java 958458f > service/src/java/org/apache/hive/service/servlet/QueryProfileServlet.java > PRE-CREATION > service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp a0b5d2e > service/src/resources/hive-webapps/static/js/bootstrap.js PRE-CREATION > service/src/resources/hive-webapps/static/js/bootstrap.min.js PRE-CREATION > service/src/resources/hive-webapps/static/js/jquery.min.js PRE-CREATION > service/src/resources/hive-webapps/static/js/tab.js PRE-CREATION > > Diff: https://reviews.apache.org/r/43008/diff/ > > > Testing > ------- > > Manual testing. Can add some unit tests in follow-up. > > > Thanks, > > Szehon Ho > >
