[SPARK-21809] Change Stage Page to use datatables to support sorting columns and searching
Support column sort, pagination and search for Stage Page using jQuery DataTable and REST API. Before this commit, the Stage page generated a hard-coded HTML table that could not support search. Supporting search and sort (over all applications rather than the 20 entries in the current page) in any case will greatly improve the user experience. Created the stagespage-template.html for displaying application information in datables. Added REST api endpoint and javascript code to fetch data from the endpoint and display it on the data table. Because of the above change, certain functionalities in the page had to be modified to support the addition of datatables. For example, the toggle checkbox 'Select All' previously would add the checked fields as columns in the Task table and as rows in the Summary Metrics table, but after the change, only columns are added in the Task Table as it got tricky to add rows dynamically in the datatables. ## How was this patch tested? I have attached the screenshots of the Stage Page UI before and after the fix. **Before:** <img width="1419" alt="30564304-35991e1c-9c8a-11e7-850f-2ac7a347f600" src="https://user-images.githubusercontent.com/22228190/42137915-52054558-7d3a-11e8-8c85-433b2c94161d.png";> <img width="1435" alt="31360592-cbaa2bae-ad14-11e7-941d-95b4c7d14970" src="https://user-images.githubusercontent.com/22228190/42137928-79df500a-7d3a-11e8-9068-5630afe46ff3.png";> **After:** <img width="1432" alt="31360591-c5650ee4-ad14-11e7-9665-5a08d8f21830" src="https://user-images.githubusercontent.com/22228190/42137936-a3fb9f42-7d3a-11e8-8502-22b3897cbf64.png";> <img width="1388" alt="31360604-d266b6b0-ad14-11e7-94b5-dcc4bb5443f4" src="https://user-images.githubusercontent.com/22228190/42137970-0fabc58c-7d3b-11e8-95ad-383b1bd1f106.png";> Closes #21688 from pgandhi999/SPARK-21809-2.3. Authored-by: pgandhi <pgan...@oath.com> Signed-off-by: Thomas Graves <tgra...@apache.org> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/76ef02e4 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/76ef02e4 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/76ef02e4 Branch: refs/heads/master Commit: 76ef02e499db49c0c6a37fa9dff3d731aeac9898 Parents: 3df307a Author: pgandhi <pgan...@oath.com> Authored: Mon Nov 26 14:08:32 2018 -0600 Committer: Thomas Graves <tgra...@apache.org> Committed: Mon Nov 26 14:08:32 2018 -0600 ---------------------------------------------------------------------- .../spark/ui/static/executorspage-template.html | 8 +- .../org/apache/spark/ui/static/executorspage.js | 84 +- .../apache/spark/ui/static/images/sort_asc.png | Bin 0 -> 160 bytes .../ui/static/images/sort_asc_disabled.png | Bin 0 -> 148 bytes .../apache/spark/ui/static/images/sort_both.png | Bin 0 -> 201 bytes .../apache/spark/ui/static/images/sort_desc.png | Bin 0 -> 158 bytes .../ui/static/images/sort_desc_disabled.png | Bin 0 -> 146 bytes .../org/apache/spark/ui/static/stagepage.js | 958 +++++++++++++ .../spark/ui/static/stagespage-template.html | 124 ++ .../org/apache/spark/ui/static/utils.js | 113 +- .../apache/spark/ui/static/webui-dataTables.css | 20 + .../org/apache/spark/ui/static/webui.css | 101 ++ .../apache/spark/status/AppStatusStore.scala | 26 +- .../spark/status/api/v1/StagesResource.scala | 121 +- .../org/apache/spark/status/api/v1/api.scala | 5 +- .../org/apache/spark/status/storeTypes.scala | 5 +- .../scala/org/apache/spark/ui/UIUtils.scala | 2 + .../apache/spark/ui/jobs/ExecutorTable.scala | 149 -- .../org/apache/spark/ui/jobs/StagePage.scala | 325 +---- .../blacklisting_for_stage_expectation.json | 1287 +++++++++--------- ...blacklisting_node_for_stage_expectation.json | 112 +- .../one_stage_attempt_json_expectation.json | 40 +- .../one_stage_json_expectation.json | 40 +- .../stage_task_list_expectation.json | 100 +- ...m_multi_attempt_app_json_1__expectation.json | 40 +- ...m_multi_attempt_app_json_2__expectation.json | 40 +- ...ask_list_w__offset___length_expectation.json | 250 +++- .../stage_task_list_w__sortBy_expectation.json | 100 +- ...ortBy_short_names___runtime_expectation.json | 100 +- ...sortBy_short_names__runtime_expectation.json | 100 +- ...stage_with_accumulable_json_expectation.json | 150 +- .../spark/status/AppStatusUtilsSuite.scala | 10 +- .../org/apache/spark/ui/StagePageSuite.scala | 12 - 33 files changed, 3064 insertions(+), 1358 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html b/core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html index 5c91304..f2c17ae 100644 --- a/core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html +++ b/core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html @@ -16,10 +16,10 @@ limitations under the License. --> <script id="executors-summary-template" type="text/html"> - <h4 style="clear: left; display: inline-block;">Summary</h4> + <h4 class="title-table">Summary</h4> <div class="container-fluid"> <div class="container-fluid"> - <table id="summary-execs-table" class="table table-striped compact"> + <table id="summary-execs-table" class="table table-striped compact cell-border"> <thead> <th></th> <th>RDD Blocks</th> @@ -64,10 +64,10 @@ limitations under the License. </table> </div> </div> - <h4 style="clear: left; display: inline-block;">Executors</h4> + <h4 class="title-table">Executors</h4> <div class="container-fluid"> <div class="container-fluid"> - <table id="active-executors-table" class="table table-striped compact"> + <table id="active-executors-table" class="table table-striped compact cell-border"> <thead> <tr> <th> http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/executorspage.js b/core/src/main/resources/org/apache/spark/ui/static/executorspage.js index 6717af3..a48c02a 100644 --- a/core/src/main/resources/org/apache/spark/ui/static/executorspage.js +++ b/core/src/main/resources/org/apache/spark/ui/static/executorspage.js @@ -59,78 +59,6 @@ $(document).ajaxStart(function () { $.blockUI({message: '<h3>Loading Executors Page...</h3>'}); }); -function createTemplateURI(appId) { - var words = document.baseURI.split('/'); - var ind = words.indexOf("proxy"); - if (ind > 0) { - var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId + '/static/executorspage-template.html'; - return baseURI; - } - ind = words.indexOf("history"); - if(ind > 0) { - var baseURI = words.slice(0, ind).join('/') + '/static/executorspage-template.html'; - return baseURI; - } - return location.origin + "/static/executorspage-template.html"; -} - -function getStandAloneppId(cb) { - var words = document.baseURI.split('/'); - var ind = words.indexOf("proxy"); - if (ind > 0) { - var appId = words[ind + 1]; - cb(appId); - return; - } - ind = words.indexOf("history"); - if (ind > 0) { - var appId = words[ind + 1]; - cb(appId); - return; - } - //Looks like Web UI is running in standalone mode - //Let's get application-id using REST End Point - $.getJSON(location.origin + "/api/v1/applications", function(response, status, jqXHR) { - if (response && response.length > 0) { - var appId = response[0].id - cb(appId); - return; - } - }); -} - -function createRESTEndPoint(appId) { - var words = document.baseURI.split('/'); - var ind = words.indexOf("proxy"); - if (ind > 0) { - var appId = words[ind + 1]; - var newBaseURI = words.slice(0, ind + 2).join('/'); - return newBaseURI + "/api/v1/applications/" + appId + "/allexecutors" - } - ind = words.indexOf("history"); - if (ind > 0) { - var appId = words[ind + 1]; - var attemptId = words[ind + 2]; - var newBaseURI = words.slice(0, ind).join('/'); - if (isNaN(attemptId)) { - return newBaseURI + "/api/v1/applications/" + appId + "/allexecutors"; - } else { - return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/allexecutors"; - } - } - return location.origin + "/api/v1/applications/" + appId + "/allexecutors"; -} - -function formatLogsCells(execLogs, type) { - if (type !== 'display') return Object.keys(execLogs); - if (!execLogs) return; - var result = ''; - $.each(execLogs, function (logName, logUrl) { - result += '<div><a href=' + logUrl + '>' + logName + '</a></div>' - }); - return result; -} - function logsExist(execs) { return execs.some(function(exec) { return !($.isEmptyObject(exec["executorLogs"])); @@ -178,17 +106,13 @@ function totalDurationColor(totalGCTime, totalDuration) { } $(document).ready(function () { - $.extend($.fn.dataTable.defaults, { - stateSave: true, - lengthMenu: [[20, 40, 60, 100, -1], [20, 40, 60, 100, "All"]], - pageLength: 20 - }); + setDataTableDefaults(); executorsSummary = $("#active-executors"); - getStandAloneppId(function (appId) { + getStandAloneAppId(function (appId) { - var endPoint = createRESTEndPoint(appId); + var endPoint = createRESTEndPointForExecutorsPage(appId); $.getJSON(endPoint, function (response, status, jqXHR) { var summary = []; var allExecCnt = 0; @@ -408,7 +332,7 @@ $(document).ready(function () { }; var data = {executors: response, "execSummary": [activeSummary, deadSummary, totalSummary]}; - $.get(createTemplateURI(appId), function (template) { + $.get(createTemplateURI(appId, "executorspage"), function (template) { executorsSummary.append(Mustache.render($(template).filter("#executors-summary-template").html(), data)); var selector = "#active-executors-table"; http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/images/sort_asc.png ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/images/sort_asc.png b/core/src/main/resources/org/apache/spark/ui/static/images/sort_asc.png new file mode 100644 index 0000000..e1ba61a Binary files /dev/null and b/core/src/main/resources/org/apache/spark/ui/static/images/sort_asc.png differ http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/images/sort_asc_disabled.png ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/images/sort_asc_disabled.png b/core/src/main/resources/org/apache/spark/ui/static/images/sort_asc_disabled.png new file mode 100644 index 0000000..fb11dfe Binary files /dev/null and b/core/src/main/resources/org/apache/spark/ui/static/images/sort_asc_disabled.png differ http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/images/sort_both.png ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/images/sort_both.png b/core/src/main/resources/org/apache/spark/ui/static/images/sort_both.png new file mode 100644 index 0000000..af5bc7c Binary files /dev/null and b/core/src/main/resources/org/apache/spark/ui/static/images/sort_both.png differ http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/images/sort_desc.png ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/images/sort_desc.png b/core/src/main/resources/org/apache/spark/ui/static/images/sort_desc.png new file mode 100644 index 0000000..0e156de Binary files /dev/null and b/core/src/main/resources/org/apache/spark/ui/static/images/sort_desc.png differ http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/images/sort_desc_disabled.png ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/images/sort_desc_disabled.png b/core/src/main/resources/org/apache/spark/ui/static/images/sort_desc_disabled.png new file mode 100644 index 0000000..c9fdd8a Binary files /dev/null and b/core/src/main/resources/org/apache/spark/ui/static/images/sort_desc_disabled.png differ http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/stagepage.js ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/stagepage.js b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js new file mode 100644 index 0000000..4c83ec7 --- /dev/null +++ b/core/src/main/resources/org/apache/spark/ui/static/stagepage.js @@ -0,0 +1,958 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +var shouldBlockUI = true; + +$(document).ajaxStop(function () { + if (shouldBlockUI) { + $.unblockUI(); + shouldBlockUI = false; + } +}); + +$(document).ajaxStart(function () { + if (shouldBlockUI) { + $.blockUI({message: '<h3>Loading Stage Page...</h3>'}); + } +}); + +$.extend( $.fn.dataTable.ext.type.order, { + "duration-pre": ConvertDurationString, + + "duration-asc": function ( a, b ) { + a = ConvertDurationString( a ); + b = ConvertDurationString( b ); + return ((a < b) ? -1 : ((a > b) ? 1 : 0)); + }, + + "duration-desc": function ( a, b ) { + a = ConvertDurationString( a ); + b = ConvertDurationString( b ); + return ((a < b) ? 1 : ((a > b) ? -1 : 0)); + } +} ); + +// This function will only parse the URL under certain format +// e.g. (history) https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4&attempt=1 +// e.g. (proxy) https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4&attempt=1 +function stageEndPoint(appId) { + var queryString = document.baseURI.split('?'); + var words = document.baseURI.split('/'); + var indexOfProxy = words.indexOf("proxy"); + var stageId = queryString[1].split("&").filter(word => word.includes("id="))[0].split("=")[1]; + if (indexOfProxy > 0) { + var appId = words[indexOfProxy + 1]; + var newBaseURI = words.slice(0, words.indexOf("proxy") + 2).join('/'); + return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId; + } + var indexOfHistory = words.indexOf("history"); + if (indexOfHistory > 0) { + var appId = words[indexOfHistory + 1]; + var appAttemptId = words[indexOfHistory + 2]; + var newBaseURI = words.slice(0, words.indexOf("history")).join('/'); + if (isNaN(appAttemptId) || appAttemptId == "0") { + return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + stageId; + } else { + return newBaseURI + "/api/v1/applications/" + appId + "/" + appAttemptId + "/stages/" + stageId; + } + } + return location.origin + "/api/v1/applications/" + appId + "/stages/" + stageId; +} + +function getColumnNameForTaskMetricSummary(columnKey) { + switch(columnKey) { + case "executorRunTime": + return "Duration"; + + case "jvmGcTime": + return "GC Time"; + + case "gettingResultTime": + return "Getting Result Time"; + + case "inputMetrics": + return "Input Size / Records"; + + case "outputMetrics": + return "Output Size / Records"; + + case "peakExecutionMemory": + return "Peak Execution Memory"; + + case "resultSerializationTime": + return "Result Serialization Time"; + + case "schedulerDelay": + return "Scheduler Delay"; + + case "diskBytesSpilled": + return "Shuffle spill (disk)"; + + case "memoryBytesSpilled": + return "Shuffle spill (memory)"; + + case "shuffleReadMetrics": + return "Shuffle Read Size / Records"; + + case "shuffleWriteMetrics": + return "Shuffle Write Size / Records"; + + case "executorDeserializeTime": + return "Task Deserialization Time"; + + case "shuffleReadBlockedTime": + return "Shuffle Read Blocked Time"; + + case "shuffleRemoteReads": + return "Shuffle Remote Reads"; + + default: + return "NA"; + } +} + +function displayRowsForSummaryMetricsTable(row, type, columnIndex) { + switch(row.columnKey) { + case 'inputMetrics': + var str = formatBytes(row.data.bytesRead[columnIndex], type) + " / " + + row.data.recordsRead[columnIndex]; + return str; + break; + + case 'outputMetrics': + var str = formatBytes(row.data.bytesWritten[columnIndex], type) + " / " + + row.data.recordsWritten[columnIndex]; + return str; + break; + + case 'shuffleReadMetrics': + var str = formatBytes(row.data.readBytes[columnIndex], type) + " / " + + row.data.readRecords[columnIndex]; + return str; + break; + + case 'shuffleReadBlockedTime': + var str = formatDuration(row.data.fetchWaitTime[columnIndex]); + return str; + break; + + case 'shuffleRemoteReads': + var str = formatBytes(row.data.remoteBytesRead[columnIndex], type); + return str; + break; + + case 'shuffleWriteMetrics': + var str = formatBytes(row.data.writeBytes[columnIndex], type) + " / " + + row.data.writeRecords[columnIndex]; + return str; + break; + + default: + return (row.columnKey == 'peakExecutionMemory' || row.columnKey == 'memoryBytesSpilled' + || row.columnKey == 'diskBytesSpilled') ? formatBytes( + row.data[columnIndex], type) : (formatDuration(row.data[columnIndex])); + + } +} + +function createDataTableForTaskSummaryMetricsTable(taskSummaryMetricsTable) { + var taskMetricsTable = "#summary-metrics-table"; + if ($.fn.dataTable.isDataTable(taskMetricsTable)) { + taskSummaryMetricsDataTable.clear().draw(); + taskSummaryMetricsDataTable.rows.add(taskSummaryMetricsTable).draw(); + } else { + var taskConf = { + "data": taskSummaryMetricsTable, + "columns": [ + {data : 'metric'}, + // Min + { + data: function (row, type) { + return displayRowsForSummaryMetricsTable(row, type, 0); + } + }, + // 25th percentile + { + data: function (row, type) { + return displayRowsForSummaryMetricsTable(row, type, 1); + } + }, + // Median + { + data: function (row, type) { + return displayRowsForSummaryMetricsTable(row, type, 2); + } + }, + // 75th percentile + { + data: function (row, type) { + return displayRowsForSummaryMetricsTable(row, type, 3); + } + }, + // Max + { + data: function (row, type) { + return displayRowsForSummaryMetricsTable(row, type, 4); + } + } + ], + "columnDefs": [ + { "type": "duration", "targets": 1 }, + { "type": "duration", "targets": 2 }, + { "type": "duration", "targets": 3 }, + { "type": "duration", "targets": 4 }, + { "type": "duration", "targets": 5 } + ], + "paging": false, + "searching": false, + "order": [[0, "asc"]], + "bSort": false, + "bAutoWidth": false + }; + taskSummaryMetricsDataTable = $(taskMetricsTable).DataTable(taskConf); + } + taskSummaryMetricsTableCurrentStateArray = taskSummaryMetricsTable.slice(); +} + +function createRowMetadataForColumn(colKey, data, checkboxId) { + var row = { + "metric": getColumnNameForTaskMetricSummary(colKey), + "data": data, + "checkboxId": checkboxId, + "columnKey": colKey + }; + return row; +} + +function reselectCheckboxesBasedOnTaskTableState() { + var allChecked = true; + var taskSummaryMetricsTableCurrentFilteredArray = taskSummaryMetricsTableCurrentStateArray.slice(); + if (typeof taskTableSelector !== 'undefined' && taskSummaryMetricsTableCurrentStateArray.length > 0) { + for (var k = 0; k < optionalColumns.length; k++) { + if (taskTableSelector.column(optionalColumns[k]).visible()) { + $("#box-"+optionalColumns[k]).prop('checked', true); + taskSummaryMetricsTableCurrentStateArray.push(taskSummaryMetricsTableArray.filter(row => (row.checkboxId).toString() == optionalColumns[k])[0]); + taskSummaryMetricsTableCurrentFilteredArray = taskSummaryMetricsTableCurrentStateArray.slice(); + } else { + allChecked = false; + } + } + if (allChecked) { + $("#box-0").prop('checked', true); + } + createDataTableForTaskSummaryMetricsTable(taskSummaryMetricsTableCurrentFilteredArray); + } +} + +function getStageAttemptId() { + var words = document.baseURI.split('?'); + var attemptIdStr = words[1].split('&')[1]; + var digitsRegex = /[0-9]+/; + // We are using regex here to extract the stage attempt id as there might be certain url's with format + // like /proxy/application_1539986433979_27115/stages/stage/?id=0&attempt=0#tasksTitle + var stgAttemptId = words[1].split("&").filter( + word => word.includes("attempt="))[0].split("=")[1].match(digitsRegex); + return stgAttemptId; +} + +var taskSummaryMetricsTableArray = []; +var taskSummaryMetricsTableCurrentStateArray = []; +var taskSummaryMetricsDataTable; +var optionalColumns = [11, 12, 13, 14, 15, 16, 17]; +var taskTableSelector; + +$(document).ready(function () { + setDataTableDefaults(); + + $("#showAdditionalMetrics").append( + "<div><a id='additionalMetrics'>" + + "<span class='expand-input-rate-arrow arrow-closed' id='arrowtoggle1'></span>" + + " Show Additional Metrics" + + "</a></div>" + + "<div class='container-fluid container-fluid-div' id='toggle-metrics' hidden>" + + "<div><input type='checkbox' class='toggle-vis' id='box-0' data-column='0'> Select All</div>" + + "<div id='scheduler_delay' class='scheduler-delay-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-11' data-column='11'> Scheduler Delay</div>" + + "<div id='task_deserialization_time' class='task-deserialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-12' data-column='12'> Task Deserialization Time</div>" + + "<div id='shuffle_read_blocked_time' class='shuffle-read-blocked-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-13' data-column='13'> Shuffle Read Blocked Time</div>" + + "<div id='shuffle_remote_reads' class='shuffle-remote-reads-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-14' data-column='14'> Shuffle Remote Reads</div>" + + "<div id='result_serialization_time' class='result-serialization-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-15' data-column='15'> Result Serialization Time</div>" + + "<div id='getting_result_time' class='getting-result-time-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-16' data-column='16'> Getting Result Time</div>" + + "<div id='peak_execution_memory' class='peak-execution-memory-checkbox-div'><input type='checkbox' class='toggle-vis' id='box-17' data-column='17'> Peak Execution Memory</div>" + + "</div>"); + + $('#scheduler_delay').attr("data-toggle", "tooltip") + .attr("data-placement", "right") + .attr("title", "Scheduler delay includes time to ship the task from the scheduler to the executor, and time to send " + + "the task result from the executor to the scheduler. If scheduler delay is large, consider decreasing the size of tasks or decreasing the size of task results."); + $('#task_deserialization_time').attr("data-toggle", "tooltip") + .attr("data-placement", "right") + .attr("title", "Time spent deserializing the task closure on the executor, including the time to read the broadcasted task."); + $('#shuffle_read_blocked_time').attr("data-toggle", "tooltip") + .attr("data-placement", "right") + .attr("title", "Time that the task spent blocked waiting for shuffle data to be read from remote machines."); + $('#shuffle_remote_reads').attr("data-toggle", "tooltip") + .attr("data-placement", "right") + .attr("title", "Total shuffle bytes read from remote executors. This is a subset of the shuffle read bytes; the remaining shuffle data is read locally. "); + $('#result_serialization_time').attr("data-toggle", "tooltip") + .attr("data-placement", "right") + .attr("title", "Time spent serializing the task result on the executor before sending it back to the driver."); + $('#getting_result_time').attr("data-toggle", "tooltip") + .attr("data-placement", "right") + .attr("title", "Time that the driver spends fetching task results from workers. If this is large, consider decreasing the amount of data returned from each task."); + $('#peak_execution_memory').attr("data-toggle", "tooltip") + .attr("data-placement", "right") + .attr("title", "Execution memory refers to the memory used by internal data structures created during " + + "shuffles, aggregations and joins when Tungsten is enabled. The value of this accumulator " + + "should be approximately the sum of the peak sizes across all such data structures created " + + "in this task. For SQL jobs, this only tracks all unsafe operators, broadcast joins, and " + + "external sort."); + $('[data-toggle="tooltip"]').tooltip(); + tasksSummary = $("#parent-container"); + getStandAloneAppId(function (appId) { + + var endPoint = stageEndPoint(appId); + var stageAttemptId = getStageAttemptId(); + $.getJSON(endPoint + "/" + stageAttemptId, function(response, status, jqXHR) { + + var responseBody = response; + var dataToShow = {}; + dataToShow.showInputData = responseBody.inputBytes > 0; + dataToShow.showOutputData = responseBody.outputBytes > 0; + dataToShow.showShuffleReadData = responseBody.shuffleReadBytes > 0; + dataToShow.showShuffleWriteData = responseBody.shuffleWriteBytes > 0; + dataToShow.showBytesSpilledData = + (responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0); + + if (!dataToShow.showShuffleReadData) { + $('#shuffle_read_blocked_time').remove(); + $('#shuffle_remote_reads').remove(); + optionalColumns.splice(2, 2); + } + + // prepare data for executor summary table + stageExecutorSummaryInfoKeys = Object.keys(responseBody.executorSummary); + $.getJSON(createRESTEndPointForExecutorsPage(appId), + function(executorSummaryResponse, status, jqXHR) { + var executorDetailsMap = {}; + executorSummaryResponse.forEach(function (executorDetail) { + executorDetailsMap[executorDetail.id] = executorDetail; + }); + + var executorSummaryTable = []; + stageExecutorSummaryInfoKeys.forEach(function (columnKeyIndex) { + var executorSummary = responseBody.executorSummary[columnKeyIndex]; + var executorDetail = executorDetailsMap[columnKeyIndex.toString()]; + executorSummary.id = columnKeyIndex; + executorSummary.executorLogs = {}; + executorSummary.hostPort = "CANNOT FIND ADDRESS"; + + if (executorDetail) { + if (executorDetail["executorLogs"]) { + responseBody.executorSummary[columnKeyIndex].executorLogs = + executorDetail["executorLogs"]; + } + if (executorDetail["hostPort"]) { + responseBody.executorSummary[columnKeyIndex].hostPort = + executorDetail["hostPort"]; + } + } + executorSummaryTable.push(responseBody.executorSummary[columnKeyIndex]); + }); + // building task aggregated metrics by executor table + var executorSummaryConf = { + "data": executorSummaryTable, + "columns": [ + {data : "id"}, + {data : "executorLogs", render: formatLogsCells}, + {data : "hostPort"}, + { + data : function (row, type) { + return type === 'display' ? formatDuration(row.taskTime) : row.taskTime; + } + }, + { + data : function (row, type) { + var totaltasks = row.succeededTasks + row.failedTasks + row.killedTasks; + return type === 'display' ? totaltasks : totaltasks.toString(); + } + }, + {data : "failedTasks"}, + {data : "killedTasks"}, + {data : "succeededTasks"}, + {data : "isBlacklistedForStage"}, + { + data : function (row, type) { + return row.inputRecords != 0 ? formatBytes(row.inputBytes, type) + " / " + row.inputRecords : ""; + } + }, + { + data : function (row, type) { + return row.outputRecords != 0 ? formatBytes(row.outputBytes, type) + " / " + row.outputRecords : ""; + } + }, + { + data : function (row, type) { + return row.shuffleReadRecords != 0 ? formatBytes(row.shuffleRead, type) + " / " + row.shuffleReadRecords : ""; + } + }, + { + data : function (row, type) { + return row.shuffleWriteRecords != 0 ? formatBytes(row.shuffleWrite, type) + " / " + row.shuffleWriteRecords : ""; + } + }, + { + data : function (row, type) { + return typeof row.memoryBytesSpilled != 'undefined' ? formatBytes(row.memoryBytesSpilled, type) : ""; + } + }, + { + data : function (row, type) { + return typeof row.diskBytesSpilled != 'undefined' ? formatBytes(row.diskBytesSpilled, type) : ""; + } + } + ], + "order": [[0, "asc"]], + "bAutoWidth": false + } + var executorSummaryTableSelector = + $("#summary-executor-table").DataTable(executorSummaryConf); + $('#parent-container [data-toggle="tooltip"]').tooltip(); + + executorSummaryTableSelector.column(9).visible(dataToShow.showInputData); + if (dataToShow.showInputData) { + $('#executor-summary-input').attr("data-toggle", "tooltip") + .attr("data-placement", "top") + .attr("title", "Bytes and records read from Hadoop or from Spark storage."); + $('#executor-summary-input').tooltip(true); + } + executorSummaryTableSelector.column(10).visible(dataToShow.showOutputData); + if (dataToShow.showOutputData) { + $('#executor-summary-output').attr("data-toggle", "tooltip") + .attr("data-placement", "top") + .attr("title", "Bytes and records written to Hadoop."); + $('#executor-summary-output').tooltip(true); + } + executorSummaryTableSelector.column(11).visible(dataToShow.showShuffleReadData); + if (dataToShow.showShuffleReadData) { + $('#executor-summary-shuffle-read').attr("data-toggle", "tooltip") + .attr("data-placement", "top") + .attr("title", "Total shuffle bytes and records read (includes both data read locally and data read from remote executors)."); + $('#executor-summary-shuffle-read').tooltip(true); + } + executorSummaryTableSelector.column(12).visible(dataToShow.showShuffleWriteData); + if (dataToShow.showShuffleWriteData) { + $('#executor-summary-shuffle-write').attr("data-toggle", "tooltip") + .attr("data-placement", "top") + .attr("title", "Bytes and records written to disk in order to be read by a shuffle in a future stage."); + $('#executor-summary-shuffle-write').tooltip(true); + } + executorSummaryTableSelector.column(13).visible(dataToShow.showBytesSpilledData); + executorSummaryTableSelector.column(14).visible(dataToShow.showBytesSpilledData); + }); + + // prepare data for accumulatorUpdates + var accumulatorTable = responseBody.accumulatorUpdates.filter(accumUpdate => + !(accumUpdate.name).toString().includes("internal.")); + + // rendering the UI page + var data = {"executors": response}; + $.get(createTemplateURI(appId, "stagespage"), function(template) { + tasksSummary.append(Mustache.render($(template).filter("#stages-summary-template").html(), data)); + + $("#additionalMetrics").click(function(){ + $("#arrowtoggle1").toggleClass("arrow-open arrow-closed"); + $("#toggle-metrics").toggle(); + if (window.localStorage) { + window.localStorage.setItem("arrowtoggle1class", $("#arrowtoggle1").attr('class')); + } + }); + + $("#aggregatedMetrics").click(function(){ + $("#arrowtoggle2").toggleClass("arrow-open arrow-closed"); + $("#toggle-aggregatedMetrics").toggle(); + if (window.localStorage) { + window.localStorage.setItem("arrowtoggle2class", $("#arrowtoggle2").attr('class')); + } + }); + + var quantiles = "0,0.25,0.5,0.75,1.0"; + $.getJSON(endPoint + "/" + stageAttemptId + "/taskSummary?quantiles=" + quantiles, + function(taskMetricsResponse, status, jqXHR) { + var taskMetricKeys = Object.keys(taskMetricsResponse); + taskMetricKeys.forEach(function (columnKey) { + switch(columnKey) { + case "shuffleReadMetrics": + var row1 = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 3); + var row2 = createRowMetadataForColumn( + "shuffleReadBlockedTime", taskMetricsResponse[columnKey], 13); + var row3 = createRowMetadataForColumn( + "shuffleRemoteReads", taskMetricsResponse[columnKey], 14); + if (dataToShow.showShuffleReadData) { + taskSummaryMetricsTableArray.push(row1); + taskSummaryMetricsTableArray.push(row2); + taskSummaryMetricsTableArray.push(row3); + } + break; + + case "schedulerDelay": + var row = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 11); + taskSummaryMetricsTableArray.push(row); + break; + + case "executorDeserializeTime": + var row = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 12); + taskSummaryMetricsTableArray.push(row); + break; + + case "resultSerializationTime": + var row = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 15); + taskSummaryMetricsTableArray.push(row); + break; + + case "gettingResultTime": + var row = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 16); + taskSummaryMetricsTableArray.push(row); + break; + + case "peakExecutionMemory": + var row = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 17); + taskSummaryMetricsTableArray.push(row); + break; + + case "inputMetrics": + var row = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 1); + if (dataToShow.showInputData) { + taskSummaryMetricsTableArray.push(row); + } + break; + + case "outputMetrics": + var row = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 2); + if (dataToShow.showOutputData) { + taskSummaryMetricsTableArray.push(row); + } + break; + + case "shuffleWriteMetrics": + var row = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 4); + if (dataToShow.showShuffleWriteData) { + taskSummaryMetricsTableArray.push(row); + } + break; + + case "diskBytesSpilled": + var row = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 5); + if (dataToShow.showBytesSpilledData) { + taskSummaryMetricsTableArray.push(row); + } + break; + + case "memoryBytesSpilled": + var row = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 6); + if (dataToShow.showBytesSpilledData) { + taskSummaryMetricsTableArray.push(row); + } + break; + + default: + if (getColumnNameForTaskMetricSummary(columnKey) != "NA") { + var row = createRowMetadataForColumn( + columnKey, taskMetricsResponse[columnKey], 0); + taskSummaryMetricsTableArray.push(row); + } + break; + } + }); + var taskSummaryMetricsTableFilteredArray = + taskSummaryMetricsTableArray.filter(row => row.checkboxId < 11); + taskSummaryMetricsTableCurrentStateArray = taskSummaryMetricsTableFilteredArray.slice(); + reselectCheckboxesBasedOnTaskTableState(); + }); + + // building accumulator update table + var accumulatorConf = { + "data": accumulatorTable, + "columns": [ + {data : "id"}, + {data : "name"}, + {data : "value"} + ], + "paging": false, + "searching": false, + "order": [[0, "asc"]], + "bAutoWidth": false + } + $("#accumulator-table").DataTable(accumulatorConf); + + // building tasks table that uses server side functionality + var totalTasksToShow = responseBody.numCompleteTasks + responseBody.numActiveTasks; + var taskTable = "#active-tasks-table"; + var taskConf = { + "serverSide": true, + "paging": true, + "info": true, + "processing": true, + "lengthMenu": [[20, 40, 60, 100, totalTasksToShow], [20, 40, 60, 100, "All"]], + "orderMulti": false, + "bAutoWidth": false, + "ajax": { + "url": endPoint + "/" + stageAttemptId + "/taskTable", + "data": function (data) { + var columnIndexToSort = 0; + var columnNameToSort = "Index"; + if (data.order[0].column && data.order[0].column != "") { + columnIndexToSort = parseInt(data.order[0].column); + columnNameToSort = data.columns[columnIndexToSort].name; + } + delete data.columns; + data.numTasks = totalTasksToShow; + data.columnIndexToSort = columnIndexToSort; + data.columnNameToSort = columnNameToSort; + }, + "dataSrc": function (jsons) { + var jsonStr = JSON.stringify(jsons); + var tasksToShow = JSON.parse(jsonStr); + return tasksToShow.aaData; + }, + "error": function (jqXHR, textStatus, errorThrown) { + alert("Unable to connect to the server. Looks like the Spark " + + "application must have ended. Please Switch to the history UI."); + $("#active-tasks-table_processing").css("display","none"); + } + }, + "columns": [ + {data: function (row, type) { + return type !== 'display' ? (isNaN(row.index) ? 0 : row.index ) : row.index; + }, + name: "Index" + }, + {data : "taskId", name: "ID"}, + {data : "attempt", name: "Attempt"}, + {data : "status", name: "Status"}, + {data : "taskLocality", name: "Locality Level"}, + {data : "executorId", name: "Executor ID"}, + {data : "host", name: "Host"}, + {data : "executorLogs", name: "Logs", render: formatLogsCells}, + {data : "launchTime", name: "Launch Time", render: formatDate}, + { + data : function (row, type) { + if (row.duration) { + return type === 'display' ? formatDuration(row.duration) : row.duration; + } else { + return ""; + } + }, + name: "Duration" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.jvmGcTime) { + return type === 'display' ? formatDuration(row.taskMetrics.jvmGcTime) : row.taskMetrics.jvmGcTime; + } else { + return ""; + } + }, + name: "GC Time" + }, + { + data : function (row, type) { + if (row.schedulerDelay) { + return type === 'display' ? formatDuration(row.schedulerDelay) : row.schedulerDelay; + } else { + return ""; + } + }, + name: "Scheduler Delay" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.executorDeserializeTime) { + return type === 'display' ? formatDuration(row.taskMetrics.executorDeserializeTime) : row.taskMetrics.executorDeserializeTime; + } else { + return ""; + } + }, + name: "Task Deserialization Time" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.shuffleReadMetrics) { + return type === 'display' ? formatDuration(row.taskMetrics.shuffleReadMetrics.fetchWaitTime) : row.taskMetrics.shuffleReadMetrics.fetchWaitTime; + } else { + return ""; + } + }, + name: "Shuffle Read Blocked Time" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.shuffleReadMetrics) { + return type === 'display' ? formatBytes(row.taskMetrics.shuffleReadMetrics.remoteBytesRead, type) : row.taskMetrics.shuffleReadMetrics.remoteBytesRead; + } else { + return ""; + } + }, + name: "Shuffle Remote Reads" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.resultSerializationTime) { + return type === 'display' ? formatDuration(row.taskMetrics.resultSerializationTime) : row.taskMetrics.resultSerializationTime; + } else { + return ""; + } + }, + name: "Result Serialization Time" + }, + { + data : function (row, type) { + if (row.gettingResultTime) { + return type === 'display' ? formatDuration(row.gettingResultTime) : row.gettingResultTime; + } else { + return ""; + } + }, + name: "Getting Result Time" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.peakExecutionMemory) { + return type === 'display' ? formatBytes(row.taskMetrics.peakExecutionMemory, type) : row.taskMetrics.peakExecutionMemory; + } else { + return ""; + } + }, + name: "Peak Execution Memory" + }, + { + data : function (row, type) { + if (accumulatorTable.length > 0 && row.accumulatorUpdates.length > 0) { + var accIndex = row.accumulatorUpdates.length - 1; + return row.accumulatorUpdates[accIndex].name + ' : ' + row.accumulatorUpdates[accIndex].update; + } else { + return ""; + } + }, + name: "Accumulators" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.inputMetrics && row.taskMetrics.inputMetrics.bytesRead > 0) { + if (type === 'display') { + return formatBytes(row.taskMetrics.inputMetrics.bytesRead, type) + " / " + row.taskMetrics.inputMetrics.recordsRead; + } else { + return row.taskMetrics.inputMetrics.bytesRead + " / " + row.taskMetrics.inputMetrics.recordsRead; + } + } else { + return ""; + } + }, + name: "Input Size / Records" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.outputMetrics && row.taskMetrics.outputMetrics.bytesWritten > 0) { + if (type === 'display') { + return formatBytes(row.taskMetrics.outputMetrics.bytesWritten, type) + " / " + row.taskMetrics.outputMetrics.recordsWritten; + } else { + return row.taskMetrics.outputMetrics.bytesWritten + " / " + row.taskMetrics.outputMetrics.recordsWritten; + } + } else { + return ""; + } + }, + name: "Output Size / Records" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.shuffleWriteMetrics && row.taskMetrics.shuffleWriteMetrics.writeTime > 0) { + return type === 'display' ? formatDuration(parseInt(row.taskMetrics.shuffleWriteMetrics.writeTime) / 1000000) : row.taskMetrics.shuffleWriteMetrics.writeTime; + } else { + return ""; + } + }, + name: "Write Time" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.shuffleWriteMetrics && row.taskMetrics.shuffleWriteMetrics.bytesWritten > 0) { + if (type === 'display') { + return formatBytes(row.taskMetrics.shuffleWriteMetrics.bytesWritten, type) + " / " + row.taskMetrics.shuffleWriteMetrics.recordsWritten; + } else { + return row.taskMetrics.shuffleWriteMetrics.bytesWritten + " / " + row.taskMetrics.shuffleWriteMetrics.recordsWritten; + } + } else { + return ""; + } + }, + name: "Shuffle Write Size / Records" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.shuffleReadMetrics && row.taskMetrics.shuffleReadMetrics.localBytesRead > 0) { + var totalBytesRead = parseInt(row.taskMetrics.shuffleReadMetrics.localBytesRead) + parseInt(row.taskMetrics.shuffleReadMetrics.remoteBytesRead); + if (type === 'display') { + return formatBytes(totalBytesRead, type) + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead; + } else { + return totalBytesRead + " / " + row.taskMetrics.shuffleReadMetrics.recordsRead; + } + } else { + return ""; + } + }, + name: "Shuffle Read Size / Records" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.memoryBytesSpilled && row.taskMetrics.memoryBytesSpilled > 0) { + return type === 'display' ? formatBytes(row.taskMetrics.memoryBytesSpilled, type) : row.taskMetrics.memoryBytesSpilled; + } else { + return ""; + } + }, + name: "Shuffle Spill (Memory)" + }, + { + data : function (row, type) { + if (row.taskMetrics && row.taskMetrics.diskBytesSpilled && row.taskMetrics.diskBytesSpilled > 0) { + return type === 'display' ? formatBytes(row.taskMetrics.diskBytesSpilled, type) : row.taskMetrics.diskBytesSpilled; + } else { + return ""; + } + }, + name: "Shuffle Spill (Disk)" + }, + { + data : function (row, type) { + var msg = row.errorMessage; + if (typeof msg === 'undefined') { + return ""; + } else { + var formHead = msg.substring(0, msg.indexOf("at")); + var form = "<span onclick=\"this.parentNode.querySelector('.stacktrace-details').classList.toggle('collapsed')\" class=\"expand-details\">+details</span>"; + var formMsg = "<div class=\"stacktrace-details collapsed\"><pre>" + row.errorMessage + "</pre></div>"; + return formHead + form + formMsg; + } + }, + name: "Errors" + } + ], + "columnDefs": [ + { "visible": false, "targets": 11 }, + { "visible": false, "targets": 12 }, + { "visible": false, "targets": 13 }, + { "visible": false, "targets": 14 }, + { "visible": false, "targets": 15 }, + { "visible": false, "targets": 16 }, + { "visible": false, "targets": 17 }, + { "visible": false, "targets": 18 } + ], + }; + taskTableSelector = $(taskTable).DataTable(taskConf); + $('#active-tasks-table_filter input').unbind(); + var searchEvent; + $('#active-tasks-table_filter input').bind('keyup', function(e) { + if (typeof searchEvent !== 'undefined') { + window.clearTimeout(searchEvent); + } + var value = this.value; + searchEvent = window.setTimeout(function(){ + taskTableSelector.search( value ).draw();}, 500); + }); + reselectCheckboxesBasedOnTaskTableState(); + + // hide or show columns dynamically event + $('input.toggle-vis').on('click', function(e){ + // Get the column + var para = $(this).attr('data-column'); + if (para == "0") { + var column = taskTableSelector.column(optionalColumns); + if ($(this).is(":checked")) { + $(".toggle-vis").prop('checked', true); + column.visible(true); + createDataTableForTaskSummaryMetricsTable(taskSummaryMetricsTableArray); + } else { + $(".toggle-vis").prop('checked', false); + column.visible(false); + var taskSummaryMetricsTableFilteredArray = + taskSummaryMetricsTableArray.filter(row => row.checkboxId < 11); + createDataTableForTaskSummaryMetricsTable(taskSummaryMetricsTableFilteredArray); + } + } else { + var column = taskTableSelector.column(para); + // Toggle the visibility + column.visible(!column.visible()); + var taskSummaryMetricsTableFilteredArray = []; + if ($(this).is(":checked")) { + taskSummaryMetricsTableCurrentStateArray.push(taskSummaryMetricsTableArray.filter(row => (row.checkboxId).toString() == para)[0]); + taskSummaryMetricsTableFilteredArray = taskSummaryMetricsTableCurrentStateArray.slice(); + } else { + taskSummaryMetricsTableFilteredArray = + taskSummaryMetricsTableCurrentStateArray.filter(row => (row.checkboxId).toString() != para); + } + createDataTableForTaskSummaryMetricsTable(taskSummaryMetricsTableFilteredArray); + } + }); + + // title number and toggle list + $("#summaryMetricsTitle").html("Summary Metrics for " + "<a href='#tasksTitle'>" + responseBody.numCompleteTasks + " Completed Tasks" + "</a>"); + $("#tasksTitle").html("Task (" + totalTasksToShow + ")"); + + // hide or show the accumulate update table + if (accumulatorTable.length == 0) { + $("#accumulator-update-table").hide(); + } else { + taskTableSelector.column(18).visible(true); + $("#accumulator-update-table").show(); + } + // Showing relevant stage data depending on stage type for task table and executor + // summary table + taskTableSelector.column(19).visible(dataToShow.showInputData); + taskTableSelector.column(20).visible(dataToShow.showOutputData); + taskTableSelector.column(21).visible(dataToShow.showShuffleWriteData); + taskTableSelector.column(22).visible(dataToShow.showShuffleWriteData); + taskTableSelector.column(23).visible(dataToShow.showShuffleReadData); + taskTableSelector.column(24).visible(dataToShow.showBytesSpilledData); + taskTableSelector.column(25).visible(dataToShow.showBytesSpilledData); + + if (window.localStorage) { + if (window.localStorage.getItem("arrowtoggle1class") !== null && + window.localStorage.getItem("arrowtoggle1class").includes("arrow-open")) { + $("#arrowtoggle1").toggleClass("arrow-open arrow-closed"); + $("#toggle-metrics").toggle(); + } + if (window.localStorage.getItem("arrowtoggle2class") !== null && + window.localStorage.getItem("arrowtoggle2class").includes("arrow-open")) { + $("#arrowtoggle2").toggleClass("arrow-open arrow-closed"); + $("#toggle-aggregatedMetrics").toggle(); + } + } + }); + }); + }); +}); http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html b/core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html new file mode 100644 index 0000000..6f950c6 --- /dev/null +++ b/core/src/main/resources/org/apache/spark/ui/static/stagespage-template.html @@ -0,0 +1,124 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to You under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +<script id="stages-summary-template" type="text/html"> + <h4 id="summaryMetricsTitle" class="title-table"></h4> + <div class="container-fluid"> + <div class="container-fluid"> + <table id="summary-metrics-table" class="table table-striped compact table-dataTable cell-border"> + <thead> + <th>Metric</th> + <th>Min</th> + <th>25th percentile</th> + <th>Median</th> + <th>75th percentile</th> + <th>Max</th> + </thead> + <tbody> + </tbody> + </table> + </div> + </div> + <a id="aggregatedMetrics"> + <span class="expand-input-rate-arrow arrow-closed" id="arrowtoggle2"></span> + <h4 class="title-table">Aggregated Metrics by Executor</h4> + </a> + <br> + <div class="container-fluid" id="toggle-aggregatedMetrics" hidden> + <div class="container-fluid"> + <table id="summary-executor-table" class="table table-striped compact table-dataTable cell-border"> + <thead> + <th>Executor ID</th> + <th>Logs</th> + <th>Address</th> + <th>Task Time</th> + <th>Total Tasks</th> + <th>Failed Tasks</th> + <th>Killed Tasks</th> + <th>Succeeded Tasks</th> + <th> + <span data-toggle="tooltip" data-placement="top" + title="Shows if this executor has been blacklisted by the scheduler due to task failures."> + Blacklisted</span> + </th> + <th><span id="executor-summary-input">Input Size / Records</span></th> + <th><span id="executor-summary-output">Output Size / Records</span></th> + <th><span id="executor-summary-shuffle-read">Shuffle Read Size / Records</span></th> + <th><span id="executor-summary-shuffle-write">Shuffle Write Size / Records</span></th> + <th>Shuffle Spill (Memory) </th> + <th>Shuffle Spill (Disk) </th> + </thead> + <tbody> + </tbody> + </table> + </div> + </div> + <div class="container-fluid" id="accumulator-update-table"> + <h4 class="title-table">Accumulators</h4> + <div class="container-fluid"> + <table id="accumulator-table" class="table table-striped compact table-dataTable cell-border"> + <thead> + <th>ID</th> + <th>Name</th> + <th>Value</th> + </thead> + <tbody> + </tbody> + </table> + </div> + </div> + <h4 id="tasksTitle" class="title-table"></h4> + <div class="container-fluid"> + <div class="container-fluid"> + <table id="active-tasks-table" class="table table-striped compact table-dataTable cell-border"> + <thead> + <tr> + <th>Index</th> + <th>Task ID</th> + <th>Attempt</th> + <th>Status</th> + <th>Locality level</th> + <th>Executor ID</th> + <th>Host</th> + <th>Logs</th> + <th>Launch Time</th> + <th>Duration</th> + <th>GC Time</th> + <th>Scheduler Delay</th> + <th>Task Deserialization Time</th> + <th>Shuffle Read Blocked Time</th> + <th>Shuffle Remote Reads</th> + <th>Result Serialization Time</th> + <th>Getting Result Time</th> + <th>Peak Execution Memory</th> + <th>Accumulators</th> + <th>Input Size / Records</th> + <th>Output Size / Records</th> + <th>Write Time</th> + <th>Shuffle Write Size / Records</th> + <th>Shuffle Read Size / Records</th> + <th>Shuffle Spill (Memory)</th> + <th>Shuffle Spill (Disk)</th> + <th>Errors</th> + </tr> + </thead> + <tbody> + </tbody> + </table> + </div> + </div> +</script> http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/utils.js ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/utils.js b/core/src/main/resources/org/apache/spark/ui/static/utils.js index 4f63f64..deeafad 100644 --- a/core/src/main/resources/org/apache/spark/ui/static/utils.js +++ b/core/src/main/resources/org/apache/spark/ui/static/utils.js @@ -18,7 +18,7 @@ // this function works exactly the same as UIUtils.formatDuration function formatDuration(milliseconds) { if (milliseconds < 100) { - return milliseconds + " ms"; + return parseInt(milliseconds).toFixed(1) + " ms"; } var seconds = milliseconds * 1.0 / 1000; if (seconds < 1) { @@ -74,3 +74,114 @@ function getTimeZone() { return new Date().toString().match(/\((.*)\)/)[1]; } } + +function formatLogsCells(execLogs, type) { + if (type !== 'display') return Object.keys(execLogs); + if (!execLogs) return; + var result = ''; + $.each(execLogs, function (logName, logUrl) { + result += '<div><a href=' + logUrl + '>' + logName + '</a></div>' + }); + return result; +} + +function getStandAloneAppId(cb) { + var words = document.baseURI.split('/'); + var ind = words.indexOf("proxy"); + if (ind > 0) { + var appId = words[ind + 1]; + cb(appId); + return; + } + ind = words.indexOf("history"); + if (ind > 0) { + var appId = words[ind + 1]; + cb(appId); + return; + } + // Looks like Web UI is running in standalone mode + // Let's get application-id using REST End Point + $.getJSON(location.origin + "/api/v1/applications", function(response, status, jqXHR) { + if (response && response.length > 0) { + var appId = response[0].id; + cb(appId); + return; + } + }); +} + +// This function is a helper function for sorting in datatable. +// When the data is in duration (e.g. 12ms 2s 2min 2h ) +// It will convert the string into integer for correct ordering +function ConvertDurationString(data) { + data = data.toString(); + var units = data.replace(/[\d\.]/g, '' ) + .replace(' ', '') + .toLowerCase(); + var multiplier = 1; + + switch(units) { + case 's': + multiplier = 1000; + break; + case 'min': + multiplier = 600000; + break; + case 'h': + multiplier = 3600000; + break; + default: + break; + } + return parseFloat(data) * multiplier; +} + +function createTemplateURI(appId, templateName) { + var words = document.baseURI.split('/'); + var ind = words.indexOf("proxy"); + if (ind > 0) { + var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId + '/static/' + templateName + '-template.html'; + return baseURI; + } + ind = words.indexOf("history"); + if(ind > 0) { + var baseURI = words.slice(0, ind).join('/') + '/static/' + templateName + '-template.html'; + return baseURI; + } + return location.origin + "/static/" + templateName + "-template.html"; +} + +function setDataTableDefaults() { + $.extend($.fn.dataTable.defaults, { + stateSave: true, + lengthMenu: [[20, 40, 60, 100, -1], [20, 40, 60, 100, "All"]], + pageLength: 20 + }); +} + +function formatDate(date) { + if (date <= 0) return "-"; + else return date.split(".")[0].replace("T", " "); +} + +function createRESTEndPointForExecutorsPage(appId) { + var words = document.baseURI.split('/'); + var ind = words.indexOf("proxy"); + if (ind > 0) { + var appId = words[ind + 1]; + var newBaseURI = words.slice(0, ind + 2).join('/'); + return newBaseURI + "/api/v1/applications/" + appId + "/allexecutors" + } + ind = words.indexOf("history"); + if (ind > 0) { + var appId = words[ind + 1]; + var attemptId = words[ind + 2]; + var newBaseURI = words.slice(0, ind).join('/'); + if (isNaN(attemptId)) { + return newBaseURI + "/api/v1/applications/" + appId + "/allexecutors"; + } else { + return newBaseURI + "/api/v1/applications/" + appId + "/" + attemptId + "/allexecutors"; + } + } + return location.origin + "/api/v1/applications/" + appId + "/allexecutors"; +} http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/webui-dataTables.css ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/webui-dataTables.css b/core/src/main/resources/org/apache/spark/ui/static/webui-dataTables.css new file mode 100644 index 0000000..f6b4abe --- /dev/null +++ b/core/src/main/resources/org/apache/spark/ui/static/webui-dataTables.css @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +table.dataTable thead .sorting_asc { background: url('images/sort_asc.png') no-repeat bottom right; } + +table.dataTable thead .sorting_desc { background: url('images/sort_desc.png') no-repeat bottom right; } \ No newline at end of file http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/resources/org/apache/spark/ui/static/webui.css ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/webui.css b/core/src/main/resources/org/apache/spark/ui/static/webui.css index 266eeec..fe5bb25 100644 --- a/core/src/main/resources/org/apache/spark/ui/static/webui.css +++ b/core/src/main/resources/org/apache/spark/ui/static/webui.css @@ -260,4 +260,105 @@ a.expandbutton { .paginate_button.active > a { color: #999999; text-decoration: underline; +} + +.title-table { + clear: left; + display: inline-block; +} + +.table-dataTable { + width: 100%; +} + +.container-fluid-div { + width: 200px; +} + +.scheduler-delay-checkbox-div { + width: 120px; +} + +.task-deserialization-time-checkbox-div { + width: 175px; +} + +.shuffle-read-blocked-time-checkbox-div { + width: 187px; +} + +.shuffle-remote-reads-checkbox-div { + width: 157px; +} + +.result-serialization-time-checkbox-div { + width: 171px; +} + +.getting-result-time-checkbox-div { + width: 141px; +} + +.peak-execution-memory-checkbox-div { + width: 170px; +} + +#active-tasks-table th { + border-top: 1px solid #dddddd; + border-bottom: 1px solid #dddddd; + border-right: 1px solid #dddddd; +} + +#active-tasks-table th:first-child { + border-left: 1px solid #dddddd; +} + +#accumulator-table th { + border-top: 1px solid #dddddd; + border-bottom: 1px solid #dddddd; + border-right: 1px solid #dddddd; +} + +#accumulator-table th:first-child { + border-left: 1px solid #dddddd; +} + +#summary-executor-table th { + border-top: 1px solid #dddddd; + border-bottom: 1px solid #dddddd; + border-right: 1px solid #dddddd; +} + +#summary-executor-table th:first-child { + border-left: 1px solid #dddddd; +} + +#summary-metrics-table th { + border-top: 1px solid #dddddd; + border-bottom: 1px solid #dddddd; + border-right: 1px solid #dddddd; +} + +#summary-metrics-table th:first-child { + border-left: 1px solid #dddddd; +} + +#summary-execs-table th { + border-top: 1px solid #dddddd; + border-bottom: 1px solid #dddddd; + border-right: 1px solid #dddddd; +} + +#summary-execs-table th:first-child { + border-left: 1px solid #dddddd; +} + +#active-executors-table th { + border-top: 1px solid #dddddd; + border-bottom: 1px solid #dddddd; + border-right: 1px solid #dddddd; +} + +#active-executors-table th:first-child { + border-left: 1px solid #dddddd; } \ No newline at end of file http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala b/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala index 63b9d89..5c0ed4d 100644 --- a/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala +++ b/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala @@ -351,7 +351,9 @@ private[spark] class AppStatusStore( def taskList(stageId: Int, stageAttemptId: Int, maxTasks: Int): Seq[v1.TaskData] = { val stageKey = Array(stageId, stageAttemptId) store.view(classOf[TaskDataWrapper]).index("stage").first(stageKey).last(stageKey).reverse() - .max(maxTasks).asScala.map(_.toApi).toSeq.reverse + .max(maxTasks).asScala.map { taskDataWrapper => + constructTaskData(taskDataWrapper) + }.toSeq.reverse } def taskList( @@ -390,7 +392,9 @@ private[spark] class AppStatusStore( } val ordered = if (ascending) indexed else indexed.reverse() - ordered.skip(offset).max(length).asScala.map(_.toApi).toSeq + ordered.skip(offset).max(length).asScala.map { taskDataWrapper => + constructTaskData(taskDataWrapper) + }.toSeq } def executorSummary(stageId: Int, attemptId: Int): Map[String, v1.ExecutorStageSummary] = { @@ -496,6 +500,24 @@ private[spark] class AppStatusStore( store.close() } + def constructTaskData(taskDataWrapper: TaskDataWrapper) : v1.TaskData = { + val taskDataOld: v1.TaskData = taskDataWrapper.toApi + val executorLogs: Option[Map[String, String]] = try { + Some(executorSummary(taskDataOld.executorId).executorLogs) + } catch { + case e: NoSuchElementException => e.getMessage + None + } + new v1.TaskData(taskDataOld.taskId, taskDataOld.index, + taskDataOld.attempt, taskDataOld.launchTime, taskDataOld.resultFetchStart, + taskDataOld.duration, taskDataOld.executorId, taskDataOld.host, taskDataOld.status, + taskDataOld.taskLocality, taskDataOld.speculative, taskDataOld.accumulatorUpdates, + taskDataOld.errorMessage, taskDataOld.taskMetrics, + executorLogs.getOrElse(Map[String, String]()), + AppStatusUtils.schedulerDelay(taskDataOld), + AppStatusUtils.gettingResultTime(taskDataOld)) + } + } private[spark] object AppStatusStore { http://git-wip-us.apache.org/repos/asf/spark/blob/76ef02e4/core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala b/core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala index 30d52b9..f818927 100644 --- a/core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala +++ b/core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala @@ -16,15 +16,16 @@ */ package org.apache.spark.status.api.v1 -import java.util.{List => JList} +import java.util.{HashMap, List => JList, Locale} import javax.ws.rs._ -import javax.ws.rs.core.MediaType +import javax.ws.rs.core.{Context, MediaType, MultivaluedMap, UriInfo} import org.apache.spark.SparkException import org.apache.spark.scheduler.StageInfo import org.apache.spark.status.api.v1.StageStatus._ import org.apache.spark.status.api.v1.TaskSorting._ import org.apache.spark.ui.SparkUI +import org.apache.spark.ui.jobs.ApiHelper._ @Produces(Array(MediaType.APPLICATION_JSON)) private[v1] class StagesResource extends BaseAppResource { @@ -102,4 +103,120 @@ private[v1] class StagesResource extends BaseAppResource { withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy)) } + // This api needs to stay formatted exactly as it is below, since, it is being used by the + // datatables for the stages page. + @GET + @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable") + def taskTable( + @PathParam("stageId") stageId: Int, + @PathParam("stageAttemptId") stageAttemptId: Int, + @QueryParam("details") @DefaultValue("true") details: Boolean, + @Context uriInfo: UriInfo): + HashMap[String, Object] = { + withUI { ui => + val uriQueryParameters = uriInfo.getQueryParameters(true) + val totalRecords = uriQueryParameters.getFirst("numTasks") + var isSearch = false + var searchValue: String = null + var filteredRecords = totalRecords + // The datatables client API sends a list of query parameters to the server which contain + // information like the columns to be sorted, search value typed by the user in the search + // box, pagination index etc. For more information on these query parameters, + // refer https://datatables.net/manual/server-side. + if (uriQueryParameters.getFirst("search[value]") != null && + uriQueryParameters.getFirst("search[value]").length > 0) { + isSearch = true + searchValue = uriQueryParameters.getFirst("search[value]") + } + val _tasksToShow: Seq[TaskData] = doPagination(uriQueryParameters, stageId, stageAttemptId, + isSearch, totalRecords.toInt) + val ret = new HashMap[String, Object]() + if (_tasksToShow.nonEmpty) { + // Performs server-side search based on input from user + if (isSearch) { + val filteredTaskList = filterTaskList(_tasksToShow, searchValue) + filteredRecords = filteredTaskList.length.toString + if (filteredTaskList.length > 0) { + val pageStartIndex = uriQueryParameters.getFirst("start").toInt + val pageLength = uriQueryParameters.getFirst("length").toInt + ret.put("aaData", filteredTaskList.slice( + pageStartIndex, pageStartIndex + pageLength)) + } else { + ret.put("aaData", filteredTaskList) + } + } else { + ret.put("aaData", _tasksToShow) + } + } else { + ret.put("aaData", _tasksToShow) + } + ret.put("recordsTotal", totalRecords) + ret.put("recordsFiltered", filteredRecords) + ret + } + } + + // Performs pagination on the server side + def doPagination(queryParameters: MultivaluedMap[String, String], stageId: Int, + stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): Seq[TaskData] = { + var columnNameToSort = queryParameters.getFirst("columnNameToSort") + // Sorting on Logs column will default to Index column sort + if (columnNameToSort.equalsIgnoreCase("Logs")) { + columnNameToSort = "Index" + } + val isAscendingStr = queryParameters.getFirst("order[0][dir]") + var pageStartIndex = 0 + var pageLength = totalRecords + // We fetch only the desired rows upto the specified page length for all cases except when a + // search query is present, in that case, we need to fetch all the rows to perform the search + // on the entire table + if (!isSearch) { + pageStartIndex = queryParameters.getFirst("start").toInt + pageLength = queryParameters.getFirst("length").toInt + } + withUI(_.store.taskList(stageId, stageAttemptId, pageStartIndex, pageLength, + indexName(columnNameToSort), isAscendingStr.equalsIgnoreCase("asc"))) + } + + // Filters task list based on search parameter + def filterTaskList( + taskDataList: Seq[TaskData], + searchValue: String): Seq[TaskData] = { + val defaultOptionString: String = "d" + val searchValueLowerCase = searchValue.toLowerCase(Locale.ROOT) + val containsValue = (taskDataParams: Any) => taskDataParams.toString.toLowerCase( + Locale.ROOT).contains(searchValueLowerCase) + val taskMetricsContainsValue = (task: TaskData) => task.taskMetrics match { + case None => false + case Some(metrics) => + (containsValue(task.taskMetrics.get.executorDeserializeTime) + || containsValue(task.taskMetrics.get.executorRunTime) + || containsValue(task.taskMetrics.get.jvmGcTime) + || containsValue(task.taskMetrics.get.resultSerializationTime) + || containsValue(task.taskMetrics.get.memoryBytesSpilled) + || containsValue(task.taskMetrics.get.diskBytesSpilled) + || containsValue(task.taskMetrics.get.peakExecutionMemory) + || containsValue(task.taskMetrics.get.inputMetrics.bytesRead) + || containsValue(task.taskMetrics.get.inputMetrics.recordsRead) + || containsValue(task.taskMetrics.get.outputMetrics.bytesWritten) + || containsValue(task.taskMetrics.get.outputMetrics.recordsWritten) + || containsValue(task.taskMetrics.get.shuffleReadMetrics.fetchWaitTime) + || containsValue(task.taskMetrics.get.shuffleReadMetrics.recordsRead) + || containsValue(task.taskMetrics.get.shuffleWriteMetrics.bytesWritten) + || containsValue(task.taskMetrics.get.shuffleWriteMetrics.recordsWritten) + || containsValue(task.taskMetrics.get.shuffleWriteMetrics.writeTime)) + } + val filteredTaskDataSequence: Seq[TaskData] = taskDataList.filter(f => + (containsValue(f.taskId) || containsValue(f.index) || containsValue(f.attempt) + || containsValue(f.launchTime) + || containsValue(f.resultFetchStart.getOrElse(defaultOptionString)) + || containsValue(f.duration.getOrElse(defaultOptionString)) + || containsValue(f.executorId) || containsValue(f.host) || containsValue(f.status) + || containsValue(f.taskLocality) || containsValue(f.speculative) + || containsValue(f.errorMessage.getOrElse(defaultOptionString)) + || taskMetricsContainsValue(f) + || containsValue(f.schedulerDelay) || containsValue(f.gettingResultTime))) + filteredTaskDataSequence + } + } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
![https://user-images.githubusercontent.com/22228190/42137915-52054558-7d3a-11e8-8c85-433b2c94161d.png"](https://user-images.githubusercontent.com/22228190/42137915-52054558-7d3a-11e8-8c85-433b2c94161d.png"){kind=link}
![https://user-images.githubusercontent.com/22228190/42137928-79df500a-7d3a-11e8-9068-5630afe46ff3.png"](https://user-images.githubusercontent.com/22228190/42137928-79df500a-7d3a-11e8-9068-5630afe46ff3.png"){kind=link}
![https://user-images.githubusercontent.com/22228190/42137936-a3fb9f42-7d3a-11e8-8502-22b3897cbf64.png"](https://user-images.githubusercontent.com/22228190/42137936-a3fb9f42-7d3a-11e8-8502-22b3897cbf64.png"){kind=link}
![https://user-images.githubusercontent.com/22228190/42137970-0fabc58c-7d3b-11e8-95ad-383b1bd1f106.png"](https://user-images.githubusercontent.com/22228190/42137970-0fabc58c-7d3b-11e8-95ad-383b1bd1f106.png"){kind=link}
