[GitHub] spark issue #22388: Revert [SPARK-24882][SQL] improve data source v2 API fro...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22388
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3027/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22388: Revert [SPARK-24882][SQL] improve data source v2 API fro...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22388
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22344: [SPARK-25352][SQL] Perform ordered global limit when lim...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22344
  
**[Test build #95966 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95966/testReport)**
 for PR 22344 at commit 
[`534e982`](https://github.com/apache/spark/commit/534e9824f6ecfa8cb04f5eb0757ff45fc448cce1).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22344: [SPARK-25352][SQL] Perform ordered global limit when lim...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22344
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3026/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22344: [SPARK-25352][SQL] Perform ordered global limit when lim...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22344
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22379
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22379
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95963/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22379
  
**[Test build #95963 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95963/testReport)**
 for PR 22379 at commit 
[`42b8227`](https://github.com/apache/spark/commit/42b82274e47541f5f8453000f8645a65ab3bdbff).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22355: [SPARK-25358][SQL] MutableProjection supports fallback t...

[sadhen]

Github user sadhen commented on the issue:

https://github.com/apache/spark/pull/22355
  
@maropu there are tips to view the codegen:

> spark.sql("explain codegen select 1 + 1").show(false)

You may explain an aggregation, and the NoOp check is obvious.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22376
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95962/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22376
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22341: [SPARK-24889][Core] Update block info when unpersist rdd...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/22341
  
Thanks @vanzin 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22383: [SPARK-25395][JavaAPI] Replace Spark Optional class with...

[mmolimar]

Github user mmolimar commented on the issue:

https://github.com/apache/spark/pull/22383
  
Done @srowen 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22395: [SPARK-16323][SQL] Add IntegralDivide expression

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/22395#discussion_r216854603
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala ---
@@ -72,6 +72,7 @@ package object dsl {
 def - (other: Expression): Expression = Subtract(expr, other)
 def * (other: Expression): Expression = Multiply(expr, other)
 def / (other: Expression): Expression = Divide(expr, other)
+def div (other: Expression): Expression = IntegralDivide(expr, other)
--- End diff --

The failure looks like relevant.
```scala
org.scalatest.exceptions.TestFailedException: 
Expected "struct<[CAST((CAST(5 AS DOUBLE) / CAST(2 AS DOUBLE)) AS 
BIGINT):big]int>",
but got "struct<[(5 div 2):]int>" Schema did not match for query #19 select 
5 div 2
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22376
  
**[Test build #95962 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95962/testReport)**
 for PR 22376 at commit 
[`4a0cffb`](https://github.com/apache/spark/commit/4a0cffb3ce9e1bede43e6a89fdd7a7b912bf93d2).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21596
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95960/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21596
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21596
  
**[Test build #95960 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95960/testReport)**
 for PR 21596 at commit 
[`f093c39`](https://github.com/apache/spark/commit/f093c39c43e5433bf0efb23fe278f5e81e55fe4c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22386: [SPARK-25399][SS] Continuous processing state sho...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/22386


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22386: [SPARK-25399][SS] Continuous processing state should not...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22386
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22386: [SPARK-25399][SS] Continuous processing state should not...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22386
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95961/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22386: [SPARK-25399][SS] Continuous processing state should not...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22386
  
**[Test build #95961 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95961/testReport)**
 for PR 22386 at commit 
[`4d4beef`](https://github.com/apache/spark/commit/4d4beef95e18d743e4b8f8bec71b7a6229cc58d3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216838598
  
--- Diff: 
core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +43,25 @@ private[v1] class StagesResource extends BaseAppResource 
{
   @PathParam("stageId") stageId: Int,
   @QueryParam("details") @DefaultValue("true") details: Boolean): 
Seq[StageData] = {
 withUI { ui =>
-  val ret = ui.store.stageData(stageId, details = details)
+  var ret = ui.store.stageData(stageId, details = details)
   if (ret.nonEmpty) {
+for (i <- 0 to (ret.length - 1)) {
--- End diff --

I would also think that things like `executorSummary.get.get(execId)` could 
be stored somewhere in the stack, e.g. clean up a bit. It's hard to follow what 
you are setting.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216814520
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -185,89 +185,9 @@ private[ui] class StagePage(parent: StagesTab, store: 
AppStatusStore) extends We
 
   
 
-val showAdditionalMetrics =
-  
-
-  
-  Show Additional Metrics
-
-
-  
-
-
-(De)select 
All
-
-
-  

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216731622
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size / Records";
+}
+else if(columnKey == "shuffleWriteMetrics") {
+return "Shuffle Write Size / Records";
+}
+else if(columnKey == "executorDeserializeTime") {
+return "Task Deserialization Time";
+}
+return "NA";
+}
+
+$(document).ready(function () {
+setDataTableDefaults();
+
+$("#showAdditionalMetrics").append(
+"" +
+   

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216483202
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://axonitered-jt1.red.ygrid.yahoo.com:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
--- End diff --

I'd replace with a switch statement.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216735579
  
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala 
---
@@ -391,6 +391,41 @@ private[spark] class AppStatusStore(
 ordered.skip(offset).max(length).asScala.map(_.toApi).toSeq
   }
 
+  // Filters task list based on search parameter
--- End diff --

if the search can happen in UI via datatables, this function should go away.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216728752
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size / Records";
+}
+else if(columnKey == "shuffleWriteMetrics") {
+return "Shuffle Write Size / Records";
+}
+else if(columnKey == "executorDeserializeTime") {
+return "Task Deserialization Time";
+}
+return "NA";
+}
+
+$(document).ready(function () {
+setDataTableDefaults();
+
+$("#showAdditionalMetrics").append(
+"" +
+   

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216484047
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://axonitered-jt1.red.ygrid.yahoo.com:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size / Records";
+}
+else if(columnKey == "shuffleWriteMetrics") {
+return "Shuffle Write Size / Records";
+}
+else if(columnKey == "executorDeserializeTime") {
+return "Task Deserialization Time";
+}
+return "NA";
+}
+
+$(document).ready(function () {
+setDataTableDefaults();
+
+$("#showAdditionalMetrics").append(
  

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216838140
  
--- Diff: 
core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala ---
@@ -40,8 +43,25 @@ private[v1] class StagesResource extends BaseAppResource 
{
   @PathParam("stageId") stageId: Int,
   @QueryParam("details") @DefaultValue("true") details: Boolean): 
Seq[StageData] = {
 withUI { ui =>
-  val ret = ui.store.stageData(stageId, details = details)
+  var ret = ui.store.stageData(stageId, details = details)
   if (ret.nonEmpty) {
+for (i <- 0 to (ret.length - 1)) {
--- End diff --

Could this be rewritten as a range loop `for (r <- ret)`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22337: [SPARK-25338][Test] Ensure to call super.beforeAll() and...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22337
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216841471
  
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -962,7 +709,7 @@ private[ui] class TaskPagedTable(
   }
 }
 
-private[ui] object ApiHelper {
+private[spark] object ApiHelper {
--- End diff --

why was this change needed?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216732491
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size / Records";
+}
+else if(columnKey == "shuffleWriteMetrics") {
+return "Shuffle Write Size / Records";
+}
+else if(columnKey == "executorDeserializeTime") {
+return "Task Deserialization Time";
+}
+return "NA";
+}
+
+$(document).ready(function () {
+setDataTableDefaults();
+
+$("#showAdditionalMetrics").append(
+"" +
+   

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216733959
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size / Records";
+}
+else if(columnKey == "shuffleWriteMetrics") {
+return "Shuffle Write Size / Records";
+}
+else if(columnKey == "executorDeserializeTime") {
+return "Task Deserialization Time";
+}
+return "NA";
+}
+
+$(document).ready(function () {
+setDataTableDefaults();
+
+$("#showAdditionalMetrics").append(
+"" +
+   

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216482853
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://axonitered-jt1.red.ygrid.yahoo.com:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
--- End diff --

I think this is an else.. it's either proxy or history.  Though I am fairly 
certain this function can (should) be rewritten as a regex. Maybe even one that 
can be configured in the future.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216733105
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size / Records";
+}
+else if(columnKey == "shuffleWriteMetrics") {
+return "Shuffle Write Size / Records";
+}
+else if(columnKey == "executorDeserializeTime") {
+return "Task Deserialization Time";
+}
+return "NA";
+}
+
+$(document).ready(function () {
+setDataTableDefaults();
+
+$("#showAdditionalMetrics").append(
+"" +
+   

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216728325
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size / Records";
+}
+else if(columnKey == "shuffleWriteMetrics") {
+return "Shuffle Write Size / Records";
+}
+else if(columnKey == "executorDeserializeTime") {
+return "Task Deserialization Time";
+}
+return "NA";
+}
+
+$(document).ready(function () {
+setDataTableDefaults();
+
+$("#showAdditionalMetrics").append(
+"" +
+   

[GitHub] spark pull request #21688: [SPARK-21809] : Change Stage Page to use datatabl...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/spark/pull/21688#discussion_r216729669
  
--- Diff: core/src/main/resources/org/apache/spark/ui/static/stagepage.js 
---
@@ -0,0 +1,807 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function () {
+$.blockUI({message: 'Loading Stage Page...'});
+});
+
+$.extend( $.fn.dataTable.ext.type.order, {
+"file-size-pre": ConvertDurationString,
+
+"file-size-asc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"file-size-desc": function ( a, b ) {
+a = ConvertDurationString( a );
+b = ConvertDurationString( b );
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+// This function will only parse the URL under certain format
+// e.g. 
https://domain:50509/history/application_1502220952225_59143/stages/stage/?id=0=0
+function stageEndPoint(appId) {
+var words = document.baseURI.split('/');
+var words2 = document.baseURI.split('?');
+var ind = words.indexOf("proxy");
+if (ind > 0) {
+var appId = words[ind + 1];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId + "/stages/" + 
stageId;
+}
+ind = words.indexOf("history");
+if (ind > 0) {
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+var newBaseURI = words.slice(0, ind).join('/');
+if (isNaN(attemptId) || attemptId == "0") {
+return newBaseURI + "/api/v1/applications/" + appId + 
"/stages/" + stageId;
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" + 
attemptId + "/stages/" + stageId;
+}
+}
+var stageIdLen = words2[1].indexOf('&');
+var stageId = words2[1].substr(3, stageIdLen - 3);
+return location.origin + "/api/v1/applications/" + appId + "/stages/" 
+ stageId;
+}
+
+function getColumnNameForTaskMetricSummary(columnKey) {
+if(columnKey == "executorRunTime") {
+return "Duration";
+}
+else if(columnKey == "jvmGcTime") {
+return "GC Time";
+}
+else if(columnKey == "gettingResultTime") {
+return "Getting Result Time";
+}
+else if(columnKey == "inputMetrics") {
+return "Input Size / Records";
+}
+else if(columnKey == "outputMetrics") {
+return "Output Size / Records";
+}
+else if(columnKey == "peakExecutionMemory") {
+return "Peak Execution Memory";
+}
+else if(columnKey == "resultSerializationTime") {
+return "Result Serialization Time";
+}
+else if(columnKey == "schedulerDelay") {
+return "Scheduler Delay";
+}
+else if(columnKey == "diskBytesSpilled") {
+return "Shuffle spill (disk)";
+}
+else if(columnKey == "memoryBytesSpilled") {
+return "Shuffle spill (memory)";
+}
+else if(columnKey == "shuffleReadMetrics") {
+return "Shuffle Read Size / Records";
+}
+else if(columnKey == "shuffleWriteMetrics") {
+return "Shuffle Write Size / Records";
+}
+else if(columnKey == "executorDeserializeTime") {
+return "Task Deserialization Time";
+}
+return "NA";
+}
+
+$(document).ready(function () {
+setDataTableDefaults();
+
+$("#showAdditionalMetrics").append(
+"" +
+   

[GitHub] spark issue #22337: [SPARK-25338][Test] Ensure to call super.beforeAll() and...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22337
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95958/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22337: [SPARK-25338][Test] Ensure to call super.beforeAll() and...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22337
  
**[Test build #95958 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95958/testReport)**
 for PR 22337 at commit 
[`2d9e34a`](https://github.com/apache/spark/commit/2d9e34abdae13efa1fac9e906f331cdd04105e82).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FlatMapGroupsWithStateSuite extends StateStoreMetricsTest `
  * `class StreamingAggregationSuite extends StateStoreMetricsTest with 
Assertions `
  * `class StreamingDeduplicationSuite extends StateStoreMetricsTest `


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13206: [SPARK-15420] [SQL] Add repartition and sort to prepare ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13206
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95964/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13206: [SPARK-15420] [SQL] Add repartition and sort to prepare ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13206
  
**[Test build #95964 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95964/consoleFull)**
 for PR 13206 at commit 
[`a64be8a`](https://github.com/apache/spark/commit/a64be8a91ddadcd7acbbd08956f214b3c40f0dca).
 * This patch **fails Spark unit tests**.
 * This patch **does not merge cleanly**.
 * This patch adds the following public classes _(experimental)_:
  * `case class DistributeAndSortOutputData(conf: CatalystConf) extends 
Rule[LogicalPlan] `


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13206: [SPARK-15420] [SQL] Add repartition and sort to prepare ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13206
  
Build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22399: [SPARK-25408] Move to mode ideomatic Java8

[TomaszGaweda]

Github user TomaszGaweda commented on the issue:

https://github.com/apache/spark/pull/22399
  
I like this idea :) +1 as it's only refactor, without logic change


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22399: [SPARK-25408] Move to mode ideomatic Java8

[TomaszGaweda]

Github user TomaszGaweda commented on a diff in the pull request:

https://github.com/apache/spark/pull/22399#discussion_r216824283
  
--- Diff: 
common/kvstore/src/test/java/org/apache/spark/util/kvstore/DBIteratorSuite.java 
---
@@ -383,7 +383,7 @@ public void testRefWithIntNaturalKey() throws Exception 
{
 LevelDBSuite.IntKeyType i = new LevelDBSuite.IntKeyType();
 i.key = 1;
 i.id = "1";
-i.values = Arrays.asList("1");
+i.values = Collections.singletonList("1");
--- End diff --

If you are already changing this, I would  also suggest to add static 
imports to shorten the code


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22364: [SPARK-25379][SQL] Improve AttributeSet and ColumnPrunin...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22364
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95957/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22364: [SPARK-25379][SQL] Improve AttributeSet and ColumnPrunin...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22364
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22364: [SPARK-25379][SQL] Improve AttributeSet and ColumnPrunin...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22364
  
**[Test build #95957 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95957/testReport)**
 for PR 22364 at commit 
[`6aeda5f`](https://github.com/apache/spark/commit/6aeda5f9c44918a3361b59594faf2b345c9dac33).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22399: [SPARK-25408] Move to mode ideomatic Java8

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22399
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22399: [SPARK-25408] Move to mode ideomatic Java8

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22399
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22399: [SPARK-25408] Move to mode ideomatic Java8

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22399
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22399: [SPARK-25408] Move to mode ideomatic Java8

[Fokko]

GitHub user Fokko opened a pull request:

https://github.com/apache/spark/pull/22399

[SPARK-25408] Move to mode ideomatic Java8

While working on another PR, I noticed that there is quite some legacy Java 
in there that can be beautified. For example the use og features from Java8, 
such as:
- Collection libraries
- Try-with-resource blocks

No code has been changed

What are your thoughts on this?

This makes code easier to read, and using try-with-resource makes is less 
likely to forget to close something.

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Fokko/spark SPARK-25408

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22399.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22399


commit 42050b540ce84063fab5d9e490e50e0eac802e94
Author: Fokko Driesprong 
Date:   2018-09-11T20:37:33Z

[SPARK-25408] Move to mode ideomatic Java8

Use features from Java8 such as:
- Collection libraries
- Try-with-resource blocks




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22379
  
**[Test build #95965 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95965/testReport)**
 for PR 22379 at commit 
[`2a0b65b`](https://github.com/apache/spark/commit/2a0b65b7774cfcfeab489795f980ed0e38d225ab).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22390: [SPARK-25402][SQL] Null handling in BooleanSimplificatio...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22390
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22390: [SPARK-25402][SQL] Null handling in BooleanSimplificatio...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22390
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95955/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22390: [SPARK-25402][SQL] Null handling in BooleanSimplificatio...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22390
  
**[Test build #95955 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95955/testReport)**
 for PR 22390 at commit 
[`b05052e`](https://github.com/apache/spark/commit/b05052e29fd1e6f8c11586c336864e8808bef737).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22379
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22379
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95948/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22398: [SPARK-23820][CORE] Enable use of long form of callsite ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22398
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22379
  
**[Test build #95948 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95948/testReport)**
 for PR 22379 at commit 
[`147c978`](https://github.com/apache/spark/commit/147c978c19cb3c976f875bf502c50fbd01140251).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22398: [SPARK-23820][CORE] Enable use of long form of callsite ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22398
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22398: [SPARK-23820][CORE] Enable use of long form of callsite ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22398
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22376
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22376
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3025/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22376
  
Kubernetes integration test status success
URL: 
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3025/



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22192
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22192
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95947/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22398: [SPARK-23820][CORE] Enable use of long form of ca...

[michaelmior]

GitHub user michaelmior opened a pull request:

https://github.com/apache/spark/pull/22398

[SPARK-23820][CORE] Enable use of long form of callsite in logs

This is a rework of #21433 to address some concerns there.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/michaelmior/spark long-callsite2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22398.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22398


commit 15edc21325f4d6cd249626032aae621880aaf75a
Author: Michael Mior 
Date:   2018-03-28T20:57:41Z

[SPARK-23820][CORE] Enable use of long form of callsite in logs




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22192
  
**[Test build #95947 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95947/testReport)**
 for PR 22192 at commit 
[`447c5e5`](https://github.com/apache/spark/commit/447c5e5974ca2a176026e63518a7a6cf29b78008).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13206: [SPARK-15420] [SQL] Add repartition and sort to prepare ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13206
  
**[Test build #95964 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95964/consoleFull)**
 for PR 13206 at commit 
[`a64be8a`](https://github.com/apache/spark/commit/a64be8a91ddadcd7acbbd08956f214b3c40f0dca).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22167: [SPARK-25170][DOC] Add list and short description...

[LucaCanali]

Github user LucaCanali closed the pull request at:

https://github.com/apache/spark/pull/22167


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22167: [SPARK-25170][DOC] Add list and short description of Spa...

[LucaCanali]

Github user LucaCanali commented on the issue:

https://github.com/apache/spark/pull/22167
  
Thanks @kiszk for reviewing this. I have addressed your comments in a new 
commit +
apologies as I have now moved this to a new PR 
https://github.com/apache/spark/pull/22397 
I am closing this to avoid confusion.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22387: [SPARK-25313][SQL][FOLLOW-UP][BACKPORT-2.3] Fix I...

[wangyum]

Github user wangyum closed the pull request at:

https://github.com/apache/spark/pull/22387


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22397: [SPARK-25170][DOC] Add list and short description of Spa...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22397
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22397: [SPARK-25170][DOC] Add list and short description of Spa...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22397
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22376
  
Kubernetes integration test starting
URL: 
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3025/



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21433: [SPARK-23820][CORE] Enable use of long form of callsite ...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/21433
  
Yea we can add this back easily.

On Tue, Sep 11, 2018 at 12:50 PM Sean Owen  wrote:

> Given lack of certainty, and that's this is small and easy to add back in
> a different form, and the fact that 2.4 is quickly teeing up, let me 
revert
> this for now. We can proceed with a different approach in a new PR.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22397: [SPARK-25170][DOC] Add list and short description of Spa...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22397
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22397: [SPARK-25170][DOC] Add list and short description...

[LucaCanali]

GitHub user LucaCanali opened a pull request:

https://github.com/apache/spark/pull/22397

[SPARK-25170][DOC] Add list and short description of Spark Executor Task 
Metrics to the documentation.

## What changes were proposed in this pull request?

Add description of Executor Task Metrics to the documentation.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/LucaCanali/spark docMonitoringTaskMetrics

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22397.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22397


commit a8db1605adbc271c785fda24b4945bf87149a4cd
Author: LucaCanali 
Date:   2018-08-20T14:12:52Z

Document Spark Executor Task Metrics

commit 27130bac74fda2a21dc1443b613c0aee4df1e17a
Author: LucaCanali 
Date:   2018-09-10T19:30:43Z

Addressing review comments.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22384: [SPARK-25398] Minor bugs from comparing unrelated...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/22384


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21433: [SPARK-23820][CORE] Enable use of long form of callsite ...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/21433
  
Given lack of certainty, and that's this is small and easy to add back in a 
different form, and the fact that 2.4 is quickly teeing up, let me revert this 
for now. We can proceed with a different approach in a new PR.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22379
  
**[Test build #95963 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95963/testReport)**
 for PR 22379 at commit 
[`42b8227`](https://github.com/apache/spark/commit/42b82274e47541f5f8453000f8645a65ab3bdbff).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22384: [SPARK-25398] Minor bugs from comparing unrelated types

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/22384
  
Merged to master/2.4


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22396: [SPARK-23425][SQL][FOLLOWUP] Support wildcards in HDFS p...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/22396
  
That's fine, and worth adding to the "Docs Text" field in SPARK-23425 as it 
will then also go in release notes. What about a quick test case for this too?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[MaxGekk]

Github user MaxGekk commented on the issue:

https://github.com/apache/spark/pull/22379
  
jenkins, retest this, please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[ifilonenko]

Github user ifilonenko commented on the issue:

https://github.com/apache/spark/pull/22376
  
This flakiness of integration tests seems in relation to the executor issue 
logged by [SPARK-25291]. Will require a separate PR to master and `2.4`. Still 
not related to this PR however. (Odd that the PRB still cannot stably run the 
full unit tests suite properly.)


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22376
  
**[Test build #95962 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95962/testReport)**
 for PR 22376 at commit 
[`4a0cffb`](https://github.com/apache/spark/commit/4a0cffb3ce9e1bede43e6a89fdd7a7b912bf93d2).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[ifilonenko]

Github user ifilonenko commented on the issue:

https://github.com/apache/spark/pull/22376
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22391: [SPARK-25371][SQL][BACKPORT-2.3] struct() should allow b...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22391
  
**[Test build #4335 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4335/testReport)**
 for PR 22391 at commit 
[`66b6bd5`](https://github.com/apache/spark/commit/66b6bd5b7e1538d60915b62fa1155dcd86f3411a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22379
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22379
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95949/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22379
  
**[Test build #95949 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95949/testReport)**
 for PR 22379 at commit 
[`42b8227`](https://github.com/apache/spark/commit/42b82274e47541f5f8453000f8645a65ab3bdbff).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22288: [SPARK-22148][SPARK-15815][Scheduler] Acquire new...

[dhruve]

Github user dhruve commented on a diff in the pull request:

https://github.com/apache/spark/pull/22288#discussion_r216795021
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -414,9 +425,48 @@ private[spark] class TaskSchedulerImpl(
 launchedAnyTask |= launchedTaskAtCurrentMaxLocality
   } while (launchedTaskAtCurrentMaxLocality)
 }
+
 if (!launchedAnyTask) {
-  taskSet.abortIfCompletelyBlacklisted(hostToExecutors)
-}
+  taskSet.getCompletelyBlacklistedTaskIfAny(hostToExecutors) match 
{
+case taskIndex: Some[Int] => // Returns the taskIndex which 
was unschedulable
+
+  // If the taskSet is unschedulable we kill an existing 
blacklisted executor/s and
+  // kick off an abortTimer which after waiting will abort the 
taskSet if we were
+  // unable to schedule any task from the taskSet.
+  // Note: We keep a track of schedulability on a per taskSet 
basis rather than on a
+  // per task basis.
+  val executor = 
hostToExecutors.valuesIterator.next().iterator.next()
--- End diff --

That's a nice suggestion. 

There was a case where you could have a few executors running, let's say 
just 3 of them and all are blacklisted but have some tasks running on them. To 
satisfy this,  I had started modifying this to take down an executor with the 
least no. of tasks running on them. I'll check some more on this.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22395: [SPARK-16323][SQL] Add IntegralDivide expression

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22395
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22395: [SPARK-16323][SQL] Add IntegralDivide expression

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22395
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95954/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22395: [SPARK-16323][SQL] Add IntegralDivide expression

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22395
  
**[Test build #95954 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95954/testReport)**
 for PR 22395 at commit 
[`649b458`](https://github.com/apache/spark/commit/649b45875e81e47ba3282150b669f766fbe806ba).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class IntegralDivide(left: Expression, right: Expression) extends 
DivModLike `


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22391: [SPARK-25371][SQL][BACKPORT-2.3] struct() should ...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/22391#discussion_r216793201
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/feature/VectorAssemblerSuite.scala ---
@@ -147,4 +147,12 @@ class VectorAssemblerSuite
   .filter(vectorUDF($"features") > 1)
   .count() == 1)
   }
+
+  test("SPARK-25371: VectorAssembler with empty inputCols") {
+val inputDF = Seq(
+  (1, Vectors.dense(1.0, 2.0)), (2, Vectors.sparse(2, Array(1), 
Array(3.0.toDF("i", "v")
+val vectorAssembler = new 
VectorAssembler().setInputCols(Array()).setOutputCol("a")
+val output = vectorAssembler.transform(inputDF)
+assert(output.select("a").limit(1).collect().head == 
Row(Vectors.sparse(0, Seq.empty)))
+  }
--- End diff --

Since `inputDF` is not important here, can we minimize the change like the 
following? The following will look more similar with the original one.
```scala
  test("SPARK-25371: VectorAssembler with empty inputCols") {
val vectorAssembler = new 
VectorAssembler().setInputCols(Array()).setOutputCol("a")
val output = vectorAssembler.transform(Seq(1).toDF("x"))
assert(output.select("a").limit(1).collect().head == 
Row(Vectors.sparse(0, Seq.empty)))
  }
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22386: [SPARK-25399][SS] Continuous processing state should not...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22386
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95959/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22386: [SPARK-25399][SS] Continuous processing state should not...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22386
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22386: [SPARK-25399][SS] Continuous processing state should not...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22386
  
**[Test build #95959 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95959/testReport)**
 for PR 22386 at commit 
[`3ebbed3`](https://github.com/apache/spark/commit/3ebbed3b5ed09638cf5f2b4e31ff28ede7bf9e73).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org