[GitHub] spark pull request #23242: [SPARK-26285][CORE] accumulator metrics sources f...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/23242#discussion_r240013209
  
--- Diff: 
examples/src/main/scala/org/apache/spark/examples/AccumulatorMetricsTest.scala 
---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples
+
+import org.apache.spark.metrics.source.{DoubleAccumulatorSource, 
LongAccumulatorSource}
+import org.apache.spark.sql.SparkSession
+
+/**
+ * Usage: AccumulatorMetricsTest [partitions] [numElem] [blockSize]
+ */
+object AccumulatorMetricsTest {
--- End diff --

@redsanket yes at first I thought that, but there are other many other 
"examples" here with the suffix Test.  


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23242: SPARK-26285: accumulator metrics sources for Long...

[abellina]

GitHub user abellina opened a pull request:

https://github.com/apache/spark/pull/23242

SPARK-26285: accumulator metrics sources for LongAccumulator and Doub…

…leAccumulator

## What changes were proposed in this pull request?

This PR implements metric sources for LongAccumulator and 
DoubleAccumulator, such that a user can register these accumulators easily and 
have their values be reported by the driver's metric namespace.

## How was this patch tested?

Unit tests, and manual tests.

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/abellina/spark SPARK-26285_accumulator_source

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/23242.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #23242


commit 45cfada1079838de90e39e488e593886b2bc33b7
Author: Alessandro Bellina 
Date:   2018-11-19T14:13:23Z

SPARK-26285: accumulator metrics sources for LongAccumulator and 
DoubleAccumulator




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...

[abellina]

Github user abellina commented on the issue:

https://github.com/apache/spark/pull/21688
  
The failing suite is `OrcQuerySuite `. Looks unrelated.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r231645553
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,960 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = true;
+
+$(document).ajaxStop(function () {
+if (blockUICount) {
+$.unblockUI();
+blockUICount = false;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+var indexOfProxy = words.indexOf("proxy");
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+if (indexOfProxy > 0) {
+var appId = words[indexOfProxy + 1];
+var newBaseURI = words.slice(0, words.indexOf("proxy") + 
2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+var indexOfHistory = words.indexOf("history");
+if (indexOfHistory > 0) {
+var appId = words[indexOfHistory + 1];
+var appAttemptId = words[indexOfHistory + 2];
+var newBaseURI = words.slice(0, 
words.indexOf("history")).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / Records";
+break;
+
+case "peakExecutionMemory":
+return "Peak Execution Memory";
+break;
+
+case "resultSerializationTime":
+return "Result Serialization Time";
+break;
+
+case "schedulerDelay":
+return "Scheduler Delay";
+break;
+
+case "diskBytesSpilled

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r231645036
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,960 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = true;
--- End diff --

since this is not a counter anymore, please rename to `shouldBlockUI` or 
something like that.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r231645490
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,960 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = true;
+
+$(document).ajaxStop(function () {
+if (blockUICount) {
+$.unblockUI();
+blockUICount = false;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+var indexOfProxy = words.indexOf("proxy");
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+if (indexOfProxy > 0) {
+var appId = words[indexOfProxy + 1];
+var newBaseURI = words.slice(0, words.indexOf("proxy") + 
2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+var indexOfHistory = words.indexOf("history");
+if (indexOfHistory > 0) {
+var appId = words[indexOfHistory + 1];
+var appAttemptId = words[indexOfHistory + 2];
+var newBaseURI = words.slice(0, 
words.indexOf("history")).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / Records";
+break;
+
+case "peakExecutionMemory":
+return "Peak Execution Memory";
+break;
+
+case "resultSerializationTime":
+return "Result Serialization Time";
+break;
+
+case "schedulerDelay":
+return "Scheduler Delay";
+break;
+
+case "diskBytesSpilled

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r230923570
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,965 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
--- End diff --

nit, I'd make this a boolean


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r231148125
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r231163762
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,965 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+var indexOfProxy = words.indexOf("proxy");
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+if (indexOfProxy > 0) {
+var appId = words[indexOfProxy + 1];
+var newBaseURI = words.slice(0, words.indexOf("proxy") + 
2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+var indexOfHistory = words.indexOf("history");
+if (indexOfHistory > 0) {
+var appId = words[indexOfHistory + 1];
+var appAttemptId = words[indexOfHistory + 2];
+var newBaseURI = words.slice(0, 
words.indexOf("history")).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / Records";
+break;
+
+case "peakExecutionMemory":
+return "Peak Execution Memory";
+break;
+
+case "resultSerializationTime":
+return "Result Serialization Time";
+break;
+
+case "schedulerDelay":
+return "Scheduler Delay";
+break;
+
+case "diskBytesSpilled

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r231147892
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r231150325
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,965 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+var indexOfProxy = words.indexOf("proxy");
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+if (indexOfProxy > 0) {
+var appId = words[indexOfProxy + 1];
+var newBaseURI = words.slice(0, words.indexOf("proxy") + 
2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+var indexOfHistory = words.indexOf("history");
+if (indexOfHistory > 0) {
+var appId = words[indexOfHistory + 1];
+var appAttemptId = words[indexOfHistory + 2];
+var newBaseURI = words.slice(0, 
words.indexOf("history")).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / Records";
+break;
+
+case "peakExecutionMemory":
+return "Peak Execution Memory";
+break;
+
+case "resultSerializationTime":
+return "Result Serialization Time";
+break;
+
+case "schedulerDelay":
+return "Scheduler Delay";
+break;
+
+case "diskBytesSpilled

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r229504900
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,959 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+var indexOfProxy = words.indexOf("proxy");
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+if (indexOfProxy > 0) {
+var appId = words[indexOfProxy + 1];
+var newBaseURI = words.slice(0, words.indexOf("proxy") + 
2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+var indexOfHistory = words.indexOf("history");
+if (indexOfHistory > 0) {
+var appId = words[indexOfHistory + 1];
+var appAttemptId = words[indexOfHistory + 2];
+var newBaseURI = words.slice(0, 
words.indexOf("history")).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / Records";
+break;
+
+case "peakExecutionMemory":
+return "Peak Execution Memory";
+break;
+
+case "resultSerializationTime":
+return "Result Serialization Time";
+break;
+
+case "schedulerDelay":
+return "Scheduler Delay";
+break;
+
+case "diskBytesSpilled

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r229438719
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,959 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+var indexOfProxy = words.indexOf("proxy");
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+if (indexOfProxy > 0) {
+var appId = words[indexOfProxy + 1];
+var newBaseURI = words.slice(0, words.indexOf("proxy") + 
2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+var indexOfHistory = words.indexOf("history");
+if (indexOfHistory > 0) {
+var appId = words[indexOfHistory + 1];
+var appAttemptId = words[indexOfHistory + 2];
+var newBaseURI = words.slice(0, 
words.indexOf("history")).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / Records";
+break;
+
+case "peakExecutionMemory":
+return "Peak Execution Memory";
+break;
+
+case "resultSerializationTime":
+return "Result Serialization Time";
+break;
+
+case "schedulerDelay":
+return "Scheduler Delay";
+break;
+
+case "diskBytesSpilled

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r229469673
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,959 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+var indexOfProxy = words.indexOf("proxy");
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+if (indexOfProxy > 0) {
+var appId = words[indexOfProxy + 1];
+var newBaseURI = words.slice(0, words.indexOf("proxy") + 
2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+var indexOfHistory = words.indexOf("history");
+if (indexOfHistory > 0) {
+var appId = words[indexOfHistory + 1];
+var appAttemptId = words[indexOfHistory + 2];
+var newBaseURI = words.slice(0, 
words.indexOf("history")).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / Records";
+break;
+
+case "peakExecutionMemory":
+return "Peak Execution Memory";
+break;
+
+case "resultSerializationTime":
+return "Result Serialization Time";
+break;
+
+case "schedulerDelay":
+return "Scheduler Delay";
+break;
+
+case "diskBytesSpilled

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r229472097
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/webui.css ---
@@ -260,4 +260,45 @@ a.expandbutton {
 .paginate_button.active > a {
 color: #99;
 text-decoration: underline;
+}
+
+.title-table {
+  clear: left;
+  display: inline-block;
+}
+
+.table-dataTable {
+  width="100%";
--- End diff --

`=` should be `:` here.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r229439305
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,959 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+var indexOfProxy = words.indexOf("proxy");
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+if (indexOfProxy > 0) {
+var appId = words[indexOfProxy + 1];
+var newBaseURI = words.slice(0, words.indexOf("proxy") + 
2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+var indexOfHistory = words.indexOf("history");
+if (indexOfHistory > 0) {
+var appId = words[indexOfHistory + 1];
+var appAttemptId = words[indexOfHistory + 2];
+var newBaseURI = words.slice(0, 
words.indexOf("history")).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / Records";
+break;
+
+case "peakExecutionMemory":
+return "Peak Execution Memory";
+break;
+
+case "resultSerializationTime":
+return "Result Serialization Time";
+break;
+
+case "schedulerDelay":
+return "Scheduler Delay";
+break;
+
+case "diskBytesSpilled

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r229478453
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/webui.css ---
@@ -260,4 +260,45 @@ a.expandbutton {
 .paginate_button.active > a {
 color: #99;
 text-decoration: underline;
+}
+
+.title-table {
+  clear: left;
+  display: inline-block;
+}
+
+.table-dataTable {
+  width="100%";
--- End diff --

and `"100%"` should be `100%`. 

Currently the tables in the stages page are not expanding to 100%.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/22288#discussion_r224879925
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -146,21 +146,31 @@ private[scheduler] class BlacklistTracker (
 nextExpiryTime = math.min(execMinExpiry, nodeMinExpiry)
   }
 
+  private def killExecutor(exec: String, msg: String): Unit = {
+allocationClient match {
+  case Some(a) =>
+logInfo(msg)
+a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, 
countFailures = false,
+  force = true)
+  case None =>
+logInfo(s"Not attempting to kill blacklisted executor id $exec " +
+  s"since allocation client is not defined.")
+}
+  }
+
   private def killBlacklistedExecutor(exec: String): Unit = {
 if (conf.get(config.BLACKLIST_KILL_ENABLED)) {
-  allocationClient match {
-case Some(a) =>
-  logInfo(s"Killing blacklisted executor id $exec " +
-s"since ${config.BLACKLIST_KILL_ENABLED.key} is set.")
-  a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, 
countFailures = false,
-force = true)
-case None =>
-  logWarning(s"Not attempting to kill blacklisted executor id 
$exec " +
-s"since allocation client is not defined.")
-  }
+  killExecutor(exec,
+s"Killing blacklisted executor id $exec since 
${config.BLACKLIST_KILL_ENABLED.key} is set.")
 }
   }
 
+  private[scheduler] def killBlacklistedIdleExecutor(exec: String): Unit = 
{
+killExecutor(exec,
--- End diff --

Makes sense. I guess there is no point in toggling this on and off (e.g. no 
IDLE_BLACKLIST_KILL_ENABLED)


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/22288#discussion_r224833138
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -146,21 +146,31 @@ private[scheduler] class BlacklistTracker (
 nextExpiryTime = math.min(execMinExpiry, nodeMinExpiry)
   }
 
+  private def killExecutor(exec: String, msg: String): Unit = {
+allocationClient match {
+  case Some(a) =>
+logInfo(msg)
+a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, 
countFailures = false,
+  force = true)
+  case None =>
+logInfo(s"Not attempting to kill blacklisted executor id $exec " +
+  s"since allocation client is not defined.")
+}
+  }
+
   private def killBlacklistedExecutor(exec: String): Unit = {
 if (conf.get(config.BLACKLIST_KILL_ENABLED)) {
-  allocationClient match {
-case Some(a) =>
-  logInfo(s"Killing blacklisted executor id $exec " +
-s"since ${config.BLACKLIST_KILL_ENABLED.key} is set.")
-  a.killExecutors(Seq(exec), adjustTargetNumExecutors = false, 
countFailures = false,
-force = true)
-case None =>
-  logWarning(s"Not attempting to kill blacklisted executor id 
$exec " +
-s"since allocation client is not defined.")
-  }
+  killExecutor(exec,
+s"Killing blacklisted executor id $exec since 
${config.BLACKLIST_KILL_ENABLED.key} is set.")
 }
   }
 
+  private[scheduler] def killBlacklistedIdleExecutor(exec: String): Unit = 
{
+killExecutor(exec,
--- End diff --

Should this code be guarded by `if 
(conf.get(config.BLACKLIST_KILL_ENABLED))`? As the other 
`killBlacklistedExecutor` function?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r224194007
  
--- Diff: 
core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,124 @@ private[v1] class StagesResource extends 
BaseAppResource {
 withUI(_.store.taskList(stageId, stageAttemptId, offset, length, 
sortBy))
   }
 
+  // This api needs to stay formatted exactly as it is below, since, it is 
being used by the
+  // datatables for the stages page.
+  @GET
+  @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+  def taskTable(
+@PathParam("stageId") stageId: Int,
+@PathParam("stageAttemptId") stageAttemptId: Int,
+@QueryParam("details") @DefaultValue("true") details: Boolean,
+@Context uriInfo: UriInfo):
+  HashMap[String, Object] = {
+withUI { ui =>
+  val uriQueryParameters = uriInfo.getQueryParameters(true)
+  val totalRecords = uriQueryParameters.getFirst("numTasks")
+  var isSearch = false
+  var searchValue: String = null
+  var filteredRecords = totalRecords
+  var _tasksToShow: Seq[TaskData] = null
+  if (uriQueryParameters.getFirst("search[value]") != null &&
+uriQueryParameters.getFirst("search[value]").length > 0) {
+_tasksToShow = doPagination(uriQueryParameters, stageId, 
stageAttemptId, true,
+  totalRecords.toInt)
+isSearch = true
+searchValue = uriQueryParameters.getFirst("search[value]")
+  } else {
+_tasksToShow = doPagination(uriQueryParameters, stageId, 
stageAttemptId, false,
+  totalRecords.toInt)
+  }
+  val ret = new HashMap[String, Object]()
+  if (_tasksToShow.nonEmpty) {
+// Performs server-side search based on input from user
+if (isSearch) {
+  val filteredTaskList = filterTaskList(_tasksToShow, searchValue)
+  filteredRecords = filteredTaskList.length.toString
+  if (filteredTaskList.length > 0) {
+val pageStartIndex = uriQueryParameters.getFirst("start").toInt
+val pageLength = uriQueryParameters.getFirst("length").toInt
+ret.put("aaData", filteredTaskList.slice(pageStartIndex, 
pageStartIndex + pageLength))
+  } else {
+ret.put("aaData", filteredTaskList)
+  }
+} else {
+  ret.put("aaData", _tasksToShow)
+}
+  } else {
+ret.put("aaData", _tasksToShow)
+  }
+  ret.put("recordsTotal", totalRecords)
+  ret.put("recordsFiltered", filteredRecords)
+  ret
+}
+  }
+
+  // Performs pagination on the server side
+  def doPagination(queryParameters: MultivaluedMap[String, String], 
stageId: Int,
+stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): 
Seq[TaskData] = {
+val queryParams = queryParameters.keySet()
+var columnToSort = 0
+if (queryParams.contains("order[0][column]")) {
+  columnToSort = queryParameters.getFirst("order[0][column]").toInt
+}
+var columnNameToSort = queryParameters.getFirst("columns[" + 
columnToSort + "][name]")
+if (columnNameToSort.equalsIgnoreCase("Logs")) {
+  columnNameToSort = "Index"
+  columnToSort = 0
+}
+val isAscendingStr = queryParameters.getFirst("order[0][dir]")
+var pageStartIndex = 0
+var pageLength = totalRecords
+if (!isSearch) {
+  pageStartIndex = queryParameters.getFirst("start").toInt
+  pageLength = queryParameters.getFirst("length").toInt
+}
+return withUI(_.store.taskList(stageId, stageAttemptId, 
pageStartIndex, pageLength,
+  indexName(columnNameToSort), isAscendingStr.equalsIgnoreCase("asc")))
+  }
+
+  // Filters task list based on search parameter
+  def filterTaskList(
+taskDataList: Seq[TaskData],
+searchValue: String): Seq[TaskData] = {
+val defaultOptionString: String = "d"
+// The task metrics dummy object below has been added to avoid 
throwing exception in cases
+// when task metrics for a particular task do not exist as of yet
+val dummyTaskMetrics: TaskMetrics = new TaskMetrics(0, 0, 0, 0, 0, 0, 
0, 0, 0, 0,
+  new InputMetrics(0, 0), new OutputMetrics(0, 0),
+  new ShuffleReadMetrics(0, 0, 0, 0, 0, 0, 0), new 
ShuffleWriteMetrics(0, 0, 0))
+val searchValueLowerCase = searchValue.toLowerCase(Loc

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r224108070
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -290,17 +210,11 @@ private[ui] class StagePage(parent: StagesTab, store: 
AppStatusStore) extends We
 desc = taskSortDesc,
 store = parent.store
   )
-  (_taskTable, _taskTable.table(page))
+  _taskTable.table(page)
--- End diff --

This call should be removed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r224106218
  
--- Diff: 
core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,124 @@ private[v1] class StagesResource extends 
BaseAppResource {
 withUI(_.store.taskList(stageId, stageAttemptId, offset, length, 
sortBy))
   }
 
+  // This api needs to stay formatted exactly as it is below, since, it is 
being used by the
+  // datatables for the stages page.
+  @GET
+  @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+  def taskTable(
+@PathParam("stageId") stageId: Int,
+@PathParam("stageAttemptId") stageAttemptId: Int,
+@QueryParam("details") @DefaultValue("true") details: Boolean,
+@Context uriInfo: UriInfo):
+  HashMap[String, Object] = {
+withUI { ui =>
+  val uriQueryParameters = uriInfo.getQueryParameters(true)
+  val totalRecords = uriQueryParameters.getFirst("numTasks")
+  var isSearch = false
+  var searchValue: String = null
+  var filteredRecords = totalRecords
+  var _tasksToShow: Seq[TaskData] = null
+  if (uriQueryParameters.getFirst("search[value]") != null &&
+uriQueryParameters.getFirst("search[value]").length > 0) {
+_tasksToShow = doPagination(uriQueryParameters, stageId, 
stageAttemptId, true,
+  totalRecords.toInt)
+isSearch = true
+searchValue = uriQueryParameters.getFirst("search[value]")
+  } else {
+_tasksToShow = doPagination(uriQueryParameters, stageId, 
stageAttemptId, false,
+  totalRecords.toInt)
+  }
+  val ret = new HashMap[String, Object]()
+  if (_tasksToShow.nonEmpty) {
+// Performs server-side search based on input from user
+if (isSearch) {
+  val filteredTaskList = filterTaskList(_tasksToShow, searchValue)
+  filteredRecords = filteredTaskList.length.toString
+  if (filteredTaskList.length > 0) {
+val pageStartIndex = uriQueryParameters.getFirst("start").toInt
+val pageLength = uriQueryParameters.getFirst("length").toInt
+ret.put("aaData", filteredTaskList.slice(pageStartIndex, 
pageStartIndex + pageLength))
+  } else {
+ret.put("aaData", filteredTaskList)
+  }
+} else {
+  ret.put("aaData", _tasksToShow)
+}
+  } else {
+ret.put("aaData", _tasksToShow)
+  }
+  ret.put("recordsTotal", totalRecords)
+  ret.put("recordsFiltered", filteredRecords)
+  ret
+}
+  }
+
+  // Performs pagination on the server side
+  def doPagination(queryParameters: MultivaluedMap[String, String], 
stageId: Int,
+stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): 
Seq[TaskData] = {
+val queryParams = queryParameters.keySet()
+var columnToSort = 0
+if (queryParams.contains("order[0][column]")) {
+  columnToSort = queryParameters.getFirst("order[0][column]").toInt
+}
+var columnNameToSort = queryParameters.getFirst("columns[" + 
columnToSort + "][name]")
+if (columnNameToSort.equalsIgnoreCase("Logs")) {
+  columnNameToSort = "Index"
+  columnToSort = 0
+}
+val isAscendingStr = queryParameters.getFirst("order[0][dir]")
+var pageStartIndex = 0
+var pageLength = totalRecords
+if (!isSearch) {
+  pageStartIndex = queryParameters.getFirst("start").toInt
+  pageLength = queryParameters.getFirst("length").toInt
+}
+return withUI(_.store.taskList(stageId, stageAttemptId, 
pageStartIndex, pageLength,
+  indexName(columnNameToSort), isAscendingStr.equalsIgnoreCase("asc")))
+  }
+
+  // Filters task list based on search parameter
+  def filterTaskList(
+taskDataList: Seq[TaskData],
+searchValue: String): Seq[TaskData] = {
+val defaultOptionString: String = "d"
+// The task metrics dummy object below has been added to avoid 
throwing exception in cases
+// when task metrics for a particular task do not exist as of yet
+val dummyTaskMetrics: TaskMetrics = new TaskMetrics(0, 0, 0, 0, 0, 0, 
0, 0, 0, 0,
+  new InputMetrics(0, 0), new OutputMetrics(0, 0),
+  new ShuffleReadMetrics(0, 0, 0, 0, 0, 0, 0), new 
ShuffleWriteMetrics(0, 0, 0))
+val searchValueLowerCase = searchValue.toLowerCase(Loc

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r223867779
  
--- Diff: 
core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -102,4 +103,159 @@ private[v1] class StagesResource extends 
BaseAppResource {
 withUI(_.store.taskList(stageId, stageAttemptId, offset, length, 
sortBy))
   }
 
+  // This api needs to stay formatted exactly as it is below, since, it is 
being used by the
+  // datatables for the stages page.
+  @GET
+  @Path("{stageId: \\d+}/{stageAttemptId: \\d+}/taskTable")
+  def taskTable(
+@PathParam("stageId") stageId: Int,
+@PathParam("stageAttemptId") stageAttemptId: Int,
+@QueryParam("details") @DefaultValue("true") details: Boolean,
+@Context uriInfo: UriInfo):
+  HashMap[String, Object] = {
+withUI { ui =>
+  val uriQueryParameters = uriInfo.getQueryParameters(true)
+  val totalRecords = uriQueryParameters.getFirst("numTasks")
+  var isSearch = false
+  var searchValue: String = null
+  var filteredRecords = totalRecords
+  var _tasksToShow: Seq[TaskData] = null
+  if (uriQueryParameters.getFirst("search[value]") != null &&
+uriQueryParameters.getFirst("search[value]").length > 0) {
+_tasksToShow = doPagination(uriQueryParameters, stageId, 
stageAttemptId, true,
+  totalRecords.toInt)
+isSearch = true
+searchValue = uriQueryParameters.getFirst("search[value]")
+  } else {
+_tasksToShow = doPagination(uriQueryParameters, stageId, 
stageAttemptId, false,
+  totalRecords.toInt)
+  }
+  val ret = new HashMap[String, Object]()
+  if (_tasksToShow.nonEmpty) {
+// Performs server-side search based on input from user
+if (isSearch) {
+  val filteredTaskList = filterTaskList(_tasksToShow, searchValue)
+  filteredRecords = filteredTaskList.length.toString
+  if (filteredTaskList.length > 0) {
+val pageStartIndex = uriQueryParameters.getFirst("start").toInt
+val pageLength = uriQueryParameters.getFirst("length").toInt
+ret.put("aaData", filteredTaskList.slice(pageStartIndex, 
pageStartIndex + pageLength))
+  } else {
+ret.put("aaData", filteredTaskList)
+  }
+} else {
+  ret.put("aaData", _tasksToShow)
+}
+  } else {
+ret.put("aaData", _tasksToShow)
+  }
+  ret.put("recordsTotal", totalRecords)
+  ret.put("recordsFiltered", filteredRecords)
+  ret
+}
+  }
+
+  // Performs pagination on the server side
+  def doPagination(queryParameters: MultivaluedMap[String, String], 
stageId: Int,
+stageAttemptId: Int, isSearch: Boolean, totalRecords: Int): 
Seq[TaskData] = {
+val queryParams = queryParameters.keySet()
+var columnToSort = 0
+if (queryParams.contains("order[0][column]")) {
+  columnToSort = queryParameters.getFirst("order[0][column]").toInt
+}
+var columnNameToSort = queryParameters.getFirst("columns[" + 
columnToSort + "][name]")
+if (columnNameToSort.equalsIgnoreCase("Logs")) {
+  columnNameToSort = "Index"
+  columnToSort = 0
+}
+val isAscendingStr = queryParameters.getFirst("order[0][dir]")
+var pageStartIndex = 0
+var pageLength = totalRecords
+if (!isSearch) {
+  pageStartIndex = queryParameters.getFirst("start").toInt
+  pageLength = queryParameters.getFirst("length").toInt
+}
+return withUI(_.store.taskList(stageId, stageAttemptId, 
pageStartIndex, pageLength,
+  indexName(columnNameToSort), isAscendingStr.equalsIgnoreCase("asc")))
+  }
+
+  // Filters task list based on search parameter
+  def filterTaskList(
+taskDataList: Seq[TaskData],
+searchValue: String): Seq[TaskData] = {
+val defaultOptionString: String = "d"
+// The task metrics dummy object below has been added to avoid 
throwing exception in cases
+// when task metrics for a particular task do not exist as of yet
+val dummyTaskMetrics: TaskMetrics = new TaskMetrics(0, 0, 0, 0, 0, 0, 
0, 0, 0, 0,
--- End diff --

I think @michal-databricks 's comment is to remove:

```
|| 
containsValue(f.taskMetrics.getOrElse(dummyTaskMetrics).executorDeserializeTime)
```
until
```

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221431470
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221431857
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221431404
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221670906
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,906 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221671401
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,906 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221431446
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221431891
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221431847
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221431384
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,926 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / Records";
+break;
+
+case "peakExecutionMemory":
+return "Peak Execution Memory";
+break;
+
+case "resultSerializationTime":
+return "Result Serialization Time";
+break;
+
+case "schedulerDelay":
+return "Scheduler Delay";
+break;
+
+case "diskBytesSpilled":
+return "Shuffle spill (disk)";
+break;
+
+case "memoryBytesSpilled":
+return "Shuffle spill (memory)";
+break;
+

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221672381
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,906 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221431683
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221431705
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r221431671
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,905 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+var blockUICount = 0;
+
+$(document).ajaxStop(function () {
+if (blockUICount == 0) {
+$.unblockUI();
+blockUICount++;
+}
+});
+
+$(document).ajaxStart(function () {
+if (blockUICount == 0) {
+$.blockUI({message: 'Loading Stage Page...'});
+}
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"duration-pre": ConvertDurationString,
+
+"duration-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"duration-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. (history) 
https://domain:50509/history/application_1536254569791_3806251/1/stages/stage/?id=4=1
+// e.g. (proxy) 
https://domain:50505/proxy/application_1502220952225_59143/stages/stage?id=4=1
+function stageEndPoint(appId) {
+var urlRegex = 
/https\:\/\/[^\/]+\/([^\/]+)\/([^\/]+)\/([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?\/?([^\/]+)?/gm;
+var urlArray = urlRegex.exec(document.baseURI);
+var ind = urlArray.indexOf("proxy");
+var queryString = document.baseURI.split('?');
+var words = document.baseURI.split('/');
+if (ind > 0) {
+var appId = urlArray[2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfProxy = words.indexOf("proxy");
+var newBaseURI = words.slice(0, indexOfProxy + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = urlArray.indexOf("history");
+if (ind > 0) {
+var appId = urlArray[2];
+var appAttemptId = urlArray[ind + 2];
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+var indexOfHistory = words.indexOf("history");
+var newBaseURI = words.slice(0, indexOfHistory).join('/');
+if (isNaN(appAttemptId) || appAttemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
appAttemptId + "/stages/" + stageId;
+}
+}
+var stageId = queryString[1].split("&").filter(word => 
word.includes("id="))[0].split("=")[1];
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+switch(columnKey) {
+case "executorRunTime":
+return "Duration";
+break;
+
+case "jvmGcTime":
+return "GC Time";
+break;
+
+case "gettingResultTime":
+return "Getting Result Time";
+break;
+
+case "inputMetrics":
+return "Input Size / Records";
+break;
+
+case "outputMetrics":
+return "Output Size / 

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r219189730
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r219189328
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size

[GitHub] spark pull request #21809: [SPARK-24851][UI] Map a Stage ID to it's Associat...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21809#discussion_r217106065
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -182,6 +197,15 @@ private[ui] class StagePage(parent: StagesTab, store: 
AppStatusStore) extends We
   {Utils.bytesToString(stageData.diskBytesSpilled)}
 
   }}
+  {if (!stageJobIds.isEmpty) {
--- End diff --

This could throw an NPE if `stageDataTuple` is `None`

```
scala> var x:Seq[Int] = null
x: Seq[Int] = null

scala> x.isEmpty
java.lang.NullPointerException
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216838598
  
--- Diff: 
core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +43,25 @@ private[v1] class StagesResource extends BaseAppResource 
{
   @PathParam("stageId") stageId: Int,
   @QueryParam("details") @DefaultValue("true") details: Boolean): 
Seq[StageData] = {
 withUI { ui =>
-  val ret = ui.store.stageData(stageId, details = details)
+  var ret = ui.store.stageData(stageId, details = details)
   if (ret.nonEmpty) {
+for (i <- 0 to (ret.length - 1)) {
--- End diff --

I would also think that things like `executorSummary.get.get(execId)` could 
be stored somewhere in the stack, e.g. clean up a bit. It's hard to follow what 
you are setting.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216814520
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -185,89 +185,9 @@ private[ui] class StagePage(parent: StagesTab, store: 
AppStatusStore) extends We
 
   
 
-val showAdditionalMetrics =
-  
-
-  
-  Show Additional Metrics
-
-
-  
-
-
-(De)select 
All
-
-
-  

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216731622
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216483202
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://axonitered-jt1.red.ygrid.yahoo.com:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
--- End diff --

I'd replace with a switch statement.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216735579
  
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala 
---
@@ -391,6 +391,41 @@ private[spark] class AppStatusStore(
 ordered.skip(offset).max(length).asScala.map(_.toApi).toSeq
   }
 
+  // Filters task list based on search parameter
--- End diff --

if the search can happen in UI via datatables, this function should go away.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216728752
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216484047
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://axonitered-jt1.red.ygrid.yahoo.com:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216838140
  
--- Diff: 
core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +43,25 @@ private[v1] class StagesResource extends BaseAppResource 
{
   @PathParam("stageId") stageId: Int,
   @QueryParam("details") @DefaultValue("true") details: Boolean): 
Seq[StageData] = {
 withUI { ui =>
-  val ret = ui.store.stageData(stageId, details = details)
+  var ret = ui.store.stageData(stageId, details = details)
   if (ret.nonEmpty) {
+for (i <- 0 to (ret.length - 1)) {
--- End diff --

Could this be rewritten as a range loop `for (r <- ret)`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216841471
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -962,7 +709,7 @@ private[ui] class TaskPagedTable(
   }
 }
 
-private[ui] object ApiHelper {
+private[spark] object ApiHelper {
--- End diff --

why was this change needed?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216732491
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216733959
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216482853
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://axonitered-jt1.red.ygrid.yahoo.com:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
--- End diff --

I think this is an else.. it's either proxy or history.  Though I am fairly 
certain this function can (should) be rewritten as a regex. Maybe even one that 
can be configured in the future.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216733105
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216728325
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216729669
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size

[GitHub] spark issue #22025: SPARK-25043: print master and appId from spark-sql on st...

[abellina]

Github user abellina commented on the issue:

https://github.com/apache/spark/pull/22025
  
@HyukjinKwon I have updated the description. Thanks for the comment.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22025: SPARK-25043: print master and appId from spark-sq...

[abellina]

GitHub user abellina opened a pull request:

https://github.com/apache/spark/pull/22025

SPARK-25043: print master and appId from spark-sql on startup

## What changes were proposed in this pull request?

A small change to print the master and appId from spark-sql as with logging 
turned down all the way, we may not know this information easily.

## How was this patch tested?

I ran spark-sql locally and saw the appId displayed as expected.

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/abellina/spark 
SPARK-25043_print_master_and_app_id_from_sparksql

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22025.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22025


commit 42c510c5486a925e570fcd77a469ea57ac5807c3
Author: Alessandro Bellina 
Date:   2018-08-07T14:16:25Z

SPARK-25043: print master and appId from spark-sql on startup




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20574: [SPARK-23385][CORE] Allow SparkUITab to be customized ad...

[abellina]

Github user abellina commented on the issue:

https://github.com/apache/spark/pull/20574
  
@LantaoLin what if all the tabs were configured this way? E.g. there is a 
default order/set of tabs, but one can move them around, or mix and match with 
custom tabs.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21512: [minor][WEB UI] Spark web ui auto refresh every x second...

[abellina]

Github user abellina commented on the issue:

https://github.com/apache/spark/pull/21512
  
@tomhanetz  when the page reload, does it jump back to Jobs each time? I 
haven't tried that, so just asking in case you know.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18253: [SPARK-18838][CORE] Introduce multiple queues in ...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/18253#discussion_r129669738
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -131,94 +132,35 @@ private[spark] class EventLoggingListener(
   }
 
   /** Log the event as JSON. */
-  private def logEvent(event: SparkListenerEvent, flushLogger: Boolean = 
false) {
+  private def logEvent(event: SparkListenerEvent) {
 val eventJson = JsonProtocol.sparkEventToJson(event)
 // scalastyle:off println
 writer.foreach(_.println(compact(render(eventJson
 // scalastyle:on println
-if (flushLogger) {
-  writer.foreach(_.flush())
-  hadoopDataStream.foreach(_.hflush())
-}
 if (testing) {
   loggedEvents += eventJson
+  flush()
 }
   }
 
-  // Events that do not trigger a flush
-  override def onStageSubmitted(event: SparkListenerStageSubmitted): Unit 
= logEvent(event)
-
-  override def onTaskStart(event: SparkListenerTaskStart): Unit = 
logEvent(event)
-
-  override def onTaskGettingResult(event: SparkListenerTaskGettingResult): 
Unit = logEvent(event)
-
-  override def onTaskEnd(event: SparkListenerTaskEnd): Unit = 
logEvent(event)
-
-  override def onEnvironmentUpdate(event: SparkListenerEnvironmentUpdate): 
Unit = {
-logEvent(redactEvent(event))
-  }
-
-  // Events that trigger a flush
-  override def onStageCompleted(event: SparkListenerStageCompleted): Unit 
= {
-logEvent(event, flushLogger = true)
+  private def flush(): Unit = {
+writer.foreach(_.flush())
+hadoopDataStream.foreach(_.hflush())
   }
 
-  override def onJobStart(event: SparkListenerJobStart): Unit = 
logEvent(event, flushLogger = true)
-
-  override def onJobEnd(event: SparkListenerJobEnd): Unit = 
logEvent(event, flushLogger = true)
-
-  override def onBlockManagerAdded(event: SparkListenerBlockManagerAdded): 
Unit = {
-logEvent(event, flushLogger = true)
-  }
-
-  override def onBlockManagerRemoved(event: 
SparkListenerBlockManagerRemoved): Unit = {
-logEvent(event, flushLogger = true)
-  }
-
-  override def onUnpersistRDD(event: SparkListenerUnpersistRDD): Unit = {
-logEvent(event, flushLogger = true)
-  }
-
-  override def onApplicationStart(event: SparkListenerApplicationStart): 
Unit = {
-logEvent(event, flushLogger = true)
-  }
-
-  override def onApplicationEnd(event: SparkListenerApplicationEnd): Unit 
= {
-logEvent(event, flushLogger = true)
-  }
-  override def onExecutorAdded(event: SparkListenerExecutorAdded): Unit = {
-logEvent(event, flushLogger = true)
-  }
-
-  override def onExecutorRemoved(event: SparkListenerExecutorRemoved): 
Unit = {
-logEvent(event, flushLogger = true)
-  }
-
-  override def onExecutorBlacklisted(event: 
SparkListenerExecutorBlacklisted): Unit = {
-logEvent(event, flushLogger = true)
-  }
-
-  override def onExecutorUnblacklisted(event: 
SparkListenerExecutorUnblacklisted): Unit = {
-logEvent(event, flushLogger = true)
-  }
-
-  override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): 
Unit = {
-logEvent(event, flushLogger = true)
-  }
-
-  override def onNodeUnblacklisted(event: SparkListenerNodeUnblacklisted): 
Unit = {
-logEvent(event, flushLogger = true)
-  }
-
-  // No-op because logging every update would be overkill
-  override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
-
-  // No-op because logging every update would be overkill
-  override def onExecutorMetricsUpdate(event: 
SparkListenerExecutorMetricsUpdate): Unit = { }
-
-  override def onOtherEvent(event: SparkListenerEvent): Unit = {
+  def log(event: SparkListenerEvent): Unit = {
 if (event.logEvent) {
-  logEvent(event, flushLogger = true)
+  val toLog = event match {
+case update: SparkListenerEnvironmentUpdate =>
+  redactEvent(update)
+case _ => event
+  }
+  logEvent(toLog)
+  nbMessageProcessed = nbMessageProcessed + 1
+  if (nbMessageProcessed == FLUSH_FREQUENCY) {
--- End diff --

This should be >= FLUSH_FREQUENCY. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.a

[GitHub] spark pull request #18253: [SPARK-18838][CORE] Introduce multiple queues in ...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/18253#discussion_r129889123
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala ---
@@ -27,7 +27,12 @@ private[spark] trait SparkListenerBus
 
   protected override def doPostEvent(
   listener: SparkListenerInterface,
-  event: SparkListenerEvent): Unit = {
+  event: SparkListenerEvent): Unit = 
SparkListenerEventDispatcher.dispatch(listener, event)
--- End diff --

why is this change necessary?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18253: [SPARK-18838][CORE] Introduce multiple queues in ...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/18253#discussion_r129659027
  
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2379,8 +2382,13 @@ class SparkContext(config: SparkConf) extends 
Logging {
 " parameter from breaking Spark's ability to find a valid 
constructor.")
   }
 }
-listenerBus.addListener(listener)
-logInfo(s"Registered listener $className")
+logInfo(s"listener $className created")
+listener
+  }
+  if (extraListeners.nonEmpty) {
+val group = new FixGroupOfListener(extraListeners, 
"extraListeners")
--- End diff --

@vanzin are you suggesting making this a call to addProcessor? Or, and 
addListener override? Just trying to understand the code at this stage.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18253: [SPARK-18838][CORE] Introduce multiple queues in ...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/18253#discussion_r129886931
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -139,30 +148,60 @@ private[spark] class LiveListenerBus(conf: SparkConf) 
extends SparkListenerBus {
   logError(s"$name has already stopped! Dropping event $event")
   return
 }
-metrics.numEventsPosted.inc()
-val eventAdded = eventQueue.offer(event)
-if (eventAdded) {
-  eventLock.release()
+defaultListenerQueue.post(event)
+otherListenerQueues.foreach(q => q.post(event))
+  }
+
+
+
+  override private[spark] def findListenersByClass[T <: 
SparkListenerInterface : ClassTag] =
+defaultListenerQueue.findListenersByClass ++ 
otherListenerQueues.flatMap(_.findListenersByClass)
+
+  override private[spark] def listeners =
+defaultListenerQueue.listeners ++ 
otherListenerQueues.flatMap(_.listeners)
+
+   /**
+* Start sending events to attached listeners.
+*
+* This first sends out all buffered events posted before this listener 
bus has started, then
+* listens for any additional events asynchronously while the listener 
bus is still running.
+* This should only be called once.
+*
+*/
+  def start(sc: SparkContext, ms: MetricsSystem): Unit = {
+startStopAddRemoveLock.lock()
+if (!started) {
+  sparkContext = sc
+  metricsSystem = ms
+  defaultListenerQueue.start(sc, ms)
+  otherListenerQueues.foreach(_.start(sc, ms))
+  started = true
 } else {
-  onDropEvent(event)
+  throw new IllegalStateException("LiveListener bus already started!")
--- End diff --

definitely want to unlock before this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18253: [SPARK-18838][CORE] Introduce multiple queues in ...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/18253#discussion_r129879433
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -39,98 +39,107 @@ import org.apache.spark.util.Utils
  * has started will events be actually propagated to all attached 
listeners. This listener bus
  * is stopped when `stop()` is called, and it will drop further events 
after stopping.
  */
-private[spark] class LiveListenerBus(conf: SparkConf) extends 
SparkListenerBus {
-
-  self =>
+private[spark] class LiveListenerBus(conf: SparkConf)
+  extends WithMultipleListenerBus[SparkListenerInterface, 
SparkListenerEvent] with Logging{
 
   import LiveListenerBus._
 
   private var sparkContext: SparkContext = _
+  private var metricsSystem: MetricsSystem = _
 
-  // Cap the capacity of the event queue so we get an explicit error 
(rather than
-  // an OOM exception) if it's perpetually being added to more quickly 
than it's being drained.
-  private val eventQueue =
-new 
LinkedBlockingQueue[SparkListenerEvent](conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY))
+  private val defaultListenerQueue =
+new GroupOfListenersBusQueue("default", 
conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY))
 
-  private[spark] val metrics = new LiveListenerBusMetrics(conf, eventQueue)
+  @volatile private var otherListenerQueues = Seq.empty[ListenerBusQueue]
 
-  // Indicate if `start()` is called
-  private val started = new AtomicBoolean(false)
-  // Indicate if `stop()` is called
+  // start, stop and add/remove listener should be mutually exclusive
+  private val startStopAddRemoveLock = new ReentrantLock()
+  // Will be set modified in a synchronized function
+  @volatile private var started = false
   private val stopped = new AtomicBoolean(false)
 
-  /** A counter for dropped events. It will be reset every time we log it. 
*/
-  private val droppedEventsCounter = new AtomicLong(0L)
-
-  /** When `droppedEventsCounter` was logged last time in milliseconds. */
-  @volatile private var lastReportTimestamp = 0L
-
-  // Indicate if we are processing some event
-  // Guarded by `self`
-  private var processingEvent = false
+   /**
+* Add a listener to the default pool.
+* This method is thread-safe and can be called in any thread.
+*/
+  final override def addListener(listener: SparkListenerInterface): Unit = 
{
+startStopAddRemoveLock.lock()
--- End diff --

same for other lock/unlocks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18253: [SPARK-18838][CORE] Introduce multiple queues in ...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/18253#discussion_r129659490
  
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -532,7 +533,10 @@ class SparkContext(config: SparkConf) extends Logging {
   new EventLoggingListener(_applicationId, _applicationAttemptId, 
_eventLogDir.get,
 _conf, _hadoopConfiguration)
 logger.start()
-listenerBus.addListener(logger)
+listenerBus.addProcessor(
--- End diff --

Found this @vanzin: 
https://github.com/apache/spark/pull/18253/files#diff-ca0fe05a42fd5edcab8a1bdaa8e58db9R86


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18253: [SPARK-18838][CORE] Introduce multiple queues in ...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/18253#discussion_r129877998
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala ---
@@ -39,98 +39,107 @@ import org.apache.spark.util.Utils
  * has started will events be actually propagated to all attached 
listeners. This listener bus
  * is stopped when `stop()` is called, and it will drop further events 
after stopping.
  */
-private[spark] class LiveListenerBus(conf: SparkConf) extends 
SparkListenerBus {
-
-  self =>
+private[spark] class LiveListenerBus(conf: SparkConf)
+  extends WithMultipleListenerBus[SparkListenerInterface, 
SparkListenerEvent] with Logging{
 
   import LiveListenerBus._
 
   private var sparkContext: SparkContext = _
+  private var metricsSystem: MetricsSystem = _
 
-  // Cap the capacity of the event queue so we get an explicit error 
(rather than
-  // an OOM exception) if it's perpetually being added to more quickly 
than it's being drained.
-  private val eventQueue =
-new 
LinkedBlockingQueue[SparkListenerEvent](conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY))
+  private val defaultListenerQueue =
+new GroupOfListenersBusQueue("default", 
conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY))
 
-  private[spark] val metrics = new LiveListenerBusMetrics(conf, eventQueue)
+  @volatile private var otherListenerQueues = Seq.empty[ListenerBusQueue]
 
-  // Indicate if `start()` is called
-  private val started = new AtomicBoolean(false)
-  // Indicate if `stop()` is called
+  // start, stop and add/remove listener should be mutually exclusive
+  private val startStopAddRemoveLock = new ReentrantLock()
+  // Will be set modified in a synchronized function
+  @volatile private var started = false
   private val stopped = new AtomicBoolean(false)
 
-  /** A counter for dropped events. It will be reset every time we log it. 
*/
-  private val droppedEventsCounter = new AtomicLong(0L)
-
-  /** When `droppedEventsCounter` was logged last time in milliseconds. */
-  @volatile private var lastReportTimestamp = 0L
-
-  // Indicate if we are processing some event
-  // Guarded by `self`
-  private var processingEvent = false
+   /**
+* Add a listener to the default pool.
+* This method is thread-safe and can be called in any thread.
+*/
+  final override def addListener(listener: SparkListenerInterface): Unit = 
{
+startStopAddRemoveLock.lock()
--- End diff --

this should probably be in a try/finally block, with unlock in the finally.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18004: [SPARK-18838][CORE] Introduce blocking strategy for Live...

[abellina]

Github user abellina commented on the issue:

https://github.com/apache/spark/pull/18004
  
So can this PR be closed now?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org