[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22967
  
**[Test build #98813 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98813/testReport)**
 for PR 22967 at commit 
[`8b6b474`](https://github.com/apache/spark/commit/8b6b4746448f99df9b42daeaf408d6d181b932e0).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22967
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5014/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[dbtsai]

Github user dbtsai commented on the issue:

https://github.com/apache/spark/pull/22967
  
@dongjoon-hyun thanks for trigging the build. The python test script was 
only looking for scala 2.11 jars resulting python test failures. I just fixed 
it in the latest push. Let's see how it goes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22967
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22967
  
**[Test build #98812 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98812/testReport)**
 for PR 22967 at commit 
[`a5cc336`](https://github.com/apache/spark/commit/a5cc336a71ad6503519c4a4210f24037d30fe4e1).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22967: [SPARK-25956] Make Scala 2.12 as default Scala ve...

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/22967#discussion_r22593
  
--- Diff: pom.xml ---
@@ -2718,7 +2710,6 @@
 
   
 *:*_2.11
-*:*_2.10
--- End diff --

Thanks for the suggestion, and I agree this will make the default scala 
2.12 profile cleaner.  


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23021: [SPARK-26032][PYTHON] Break large sql/tests.py fi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/23021


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/23021
  
Merged to master.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/23021
  
I am merging this in - maybe I am rushing it but please allow me to go 
ahead since it's going to block other PySpark PRs.

At worst case, I am willing to revert and propose this again if there's 
something obviously wrong. Please feel free to leave some comments even after 
this is merged. I would appreciate it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/23012
  
Ah .. right makes sense to me. Thanks @shaneknapp. +1


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22966: [SPARK-25965][SQL][TEST] Add avro read benchmark

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22966
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98811/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22966: [SPARK-25965][SQL][TEST] Add avro read benchmark

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22966
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22966: [SPARK-25965][SQL][TEST] Add avro read benchmark

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22966
  
**[Test build #98811 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98811/testReport)**
 for PR 22966 at commit 
[`e6b73f1`](https://github.com/apache/spark/commit/e6b73f120b784cc548505e70802b8ec821e4a04b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23029: [SPARK-26055][CORE] InterfaceStability annotation...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/23029#discussion_r233322079
  
--- Diff: 
common/tags/src/main/java/org/apache/spark/annotation/InterfaceStability.java 
---
@@ -17,7 +17,7 @@
 
 package org.apache.spark.annotation;
 
-import java.lang.annotation.Documented;
+import java.lang.annotation.*;
--- End diff --

Actually we need to import 5 classes, that's why my IDE turned it into * 
automatically...


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22966: [SPARK-25965][SQL][TEST] Add avro read benchmark

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22966
  
**[Test build #98811 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98811/testReport)**
 for PR 22966 at commit 
[`e6b73f1`](https://github.com/apache/spark/commit/e6b73f120b784cc548505e70802b8ec821e4a04b).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23029: [SPARK-26055][CORE] InterfaceStability annotation...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/23029#discussion_r233321272
  
--- Diff: 
common/tags/src/main/java/org/apache/spark/annotation/InterfaceStability.java 
---
@@ -17,7 +17,7 @@
 
 package org.apache.spark.annotation;
 
-import java.lang.annotation.Documented;
+import java.lang.annotation.*;
--- End diff --

No big deal, but I think we'd usually explicitly import 3 classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23029: [SPARK-26055][CORE] InterfaceStability annotations shoul...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23029
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5012/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22966: [SPARK-25965][SQL][TEST] Add avro read benchmark

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22966
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5013/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23029: [SPARK-26055][CORE] InterfaceStability annotations shoul...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23029
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22966: [SPARK-25965][SQL][TEST] Add avro read benchmark

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22966
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22966: [SPARK-25965][SQL][TEST] Add avro read benchmark

[gengliangwang]

Github user gengliangwang commented on the issue:

https://github.com/apache/spark/pull/22966
  
@dongjoon-hyun sure.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23029: [SPARK-26055][CORE] InterfaceStability annotations shoul...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23029
  
**[Test build #98810 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98810/testReport)**
 for PR 23029 at commit 
[`b71dd6d`](https://github.com/apache/spark/commit/b71dd6d86378a086ea66915b4041d844248eaacb).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23029: [SPARK-26055][CORE] InterfaceStability annotation...

[cloud-fan]

GitHub user cloud-fan opened a pull request:

https://github.com/apache/spark/pull/23029

[SPARK-26055][CORE] InterfaceStability annotations should be retained at 
runtime

## What changes were proposed in this pull request?

It's good to have annotations available at runtime, so that tools like MiMa 
can detect them and deal with then specially. e.g. we don't want to track 
compatibility for unstable classes.

This PR makes `InterfaceStability` annotations to be retained at runtime, 
to be consistent with `Experimental` and `DeveloperApi`

## How was this patch tested?
N/A


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloud-fan/spark annotation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/23029.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #23029


commit b71dd6d86378a086ea66915b4041d844248eaacb
Author: Wenchen Fan 
Date:   2018-11-14T05:39:00Z

InterfaceStability annotations should be retained at runtime




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23029: [SPARK-26055][CORE] InterfaceStability annotations shoul...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/23029
  
cc @rxin @srowen @vanzin @gatorsmile 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22976: [SPARK-25974][SQL]Optimizes Generates bytecode for order...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/22976
  
LGTM


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23021
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23021
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98808/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23021
  
**[Test build #98808 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98808/testReport)**
 for PR 23021 at commit 
[`a990770`](https://github.com/apache/spark/commit/a990770ca17e49ac6a04fa5f3e62455af5ef7170).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22867: [SPARK-25778] WriteAheadLogBackedBlockRDD in YARN Cluste...

[gss2002]

Github user gss2002 commented on the issue:

https://github.com/apache/spark/pull/22867
  
@vanzin can you review the latest PR. This seems like it works without 
issue. I ran this on my home cluster and my larger clusters at work where the 
issue was first discovered and it seems to solve it. Thanks again for the help


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22989: [SPARK-25986][Build] Add rules to ban throw Error...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/22989#discussion_r233318346
  
--- Diff: dev/checkstyle.xml ---
@@ -180,5 +180,10 @@
 
 
 
+
+
+
--- End diff --

What `application code` means here?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23028
  
**[Test build #98809 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98809/testReport)**
 for PR 23028 at commit 
[`56a02ea`](https://github.com/apache/spark/commit/56a02eaaa63f297d3dbaf0ca183e4248d4882834).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23028
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23028
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98809/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22989: [SPARK-25986][Build] Add rules to ban throw Error...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/22989#discussion_r233317742
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/mllib/clustering/KMeansSuite.scala ---
@@ -331,7 +333,7 @@ class KMeansSuite extends SparkFunSuite with 
MLlibTestSparkContext {
   }
 }
 
-object KMeansSuite extends SparkFunSuite {
+object KMeansSuite extends SparkFunSuite with Assertions {
--- End diff --

Isn't `Assertions` redundant?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[wangyum]

Github user wangyum commented on a diff in the pull request:

https://github.com/apache/spark/pull/23028#discussion_r233317201
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
 ---
@@ -504,6 +504,19 @@ object LikeSimplification extends Rule[LogicalPlan] {
 Like(input, Literal.create(pattern, StringType))
 }
   }
+
+case Like(Literal(pattern, StringType), input) =>
+  if (pattern == null) {
+// If pattern is null, return null value directly, since "null 
like col" == null.
+Literal(null, BooleanType)
+  } else {
+pattern.toString match {
+  case equalTo(str) =>
+EqualTo(Literal(str), input)
--- End diff --

Yes. 
```
select  "abc" like "%abc%" -> true
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[wangyum]

Github user wangyum closed the pull request at:

https://github.com/apache/spark/pull/23028


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/23028#discussion_r233315756
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
 ---
@@ -504,6 +504,19 @@ object LikeSimplification extends Rule[LogicalPlan] {
 Like(input, Literal.create(pattern, StringType))
 }
   }
+
+case Like(Literal(pattern, StringType), input) =>
+  if (pattern == null) {
+// If pattern is null, return null value directly, since "null 
like col" == null.
+Literal(null, BooleanType)
+  } else {
+pattern.toString match {
+  case equalTo(str) =>
+EqualTo(Literal(str), input)
--- End diff --

This is not correct. E.g., "abc" like "abc%"?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/23028#discussion_r233315067
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
 ---
@@ -504,6 +504,19 @@ object LikeSimplification extends Rule[LogicalPlan] {
 Like(input, Literal.create(pattern, StringType))
 }
   }
+
+case Like(Literal(pattern, StringType), input) =>
--- End diff --

Left input to `Like` is string input. This makes confusing.  Right 
parameter is `pattern`.

```scala
case Like(Literal(input, StringType), pattern) =>
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/23028#discussion_r233315192
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
 ---
@@ -504,6 +504,19 @@ object LikeSimplification extends Rule[LogicalPlan] {
 Like(input, Literal.create(pattern, StringType))
 }
   }
+
+case Like(Literal(pattern, StringType), input) =>
+  if (pattern == null) {
+// If pattern is null, return null value directly, since "null 
like col" == null.
--- End diff --

This is rather like `If string input is null, return null value directly, 
... `.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22967
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98807/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22967
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22967
  
**[Test build #98807 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98807/testReport)**
 for PR 22967 at commit 
[`52dc4a1`](https://github.com/apache/spark/commit/52dc4a1d625154fb3baab201f9ff3f979b497602).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Client mode...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22504
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Client mode...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22504
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98805/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Client mode...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22504
  
**[Test build #98805 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98805/testReport)**
 for PR 22504 at commit 
[`616bd01`](https://github.com/apache/spark/commit/616bd01b5b23dc0ce0b50e95b7fd6684b1f7e9d5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...

[wangyum]

Github user wangyum commented on the issue:

https://github.com/apache/spark/pull/23027
  
cc @HyukjinKwon @dongjoon-hyun 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23027
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23027
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98806/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23027
  
**[Test build #98806 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98806/testReport)**
 for PR 23027 at commit 
[`d0a2a3f`](https://github.com/apache/spark/commit/d0a2a3f4af492fbf69f7774e03d67d4af39cd5c7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23028
  
**[Test build #98809 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98809/testReport)**
 for PR 23028 at commit 
[`56a02ea`](https://github.com/apache/spark/commit/56a02eaaa63f297d3dbaf0ca183e4248d4882834).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23028
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23028
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5011/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23028: [SPARK-26053][SQL] Enhance LikeSimplification

[wangyum]

GitHub user wangyum opened a pull request:

https://github.com/apache/spark/pull/23028

[SPARK-26053][SQL] Enhance LikeSimplification

## What changes were proposed in this pull request?

This PR enhance `LikeSimplification` in 2 cases:

1.  null like col -> null
2. 'str' like col -> col = 'str'

It difficult to handle these cases:
1. 'str%' like col
2. '%str' like col
3. 'str%str' like col
4. '%' like col

for example:
```sql
select '8%' like '8%';  -- true
select '8%' like '%8%';  -- true
select '8%' like '%%8%%';  -- true
select '8%' like '%%5%%8%%'; --false

select '%8' like '%8%';  -- true
select '%8' like '%8%';  -- true
select '%8' like '%%8%';  -- true
select '%8' like '%%5%%8%'; -- false

select '%' like '%';  -- true
select '%' like '%%';  -- true
select '%' like '%%8%';  -- false
```

## How was this patch tested?

unit tests


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wangyum/spark SPARK-26053

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/23028.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #23028


commit 56a02eaaa63f297d3dbaf0ca183e4248d4882834
Author: Yuming Wang 
Date:   2018-11-14T02:55:25Z

Enhance LikeSimplification




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22993: [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cleaner in...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22993
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98804/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22993: [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cleaner in...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22993
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22993: [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cleaner in...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22993
  
**[Test build #98804 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98804/testReport)**
 for PR 22993 at commit 
[`2ee58b2`](https://github.com/apache/spark/commit/2ee58b2ba66fbf497e819feff9beb786d4884acd).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/23012
  
TL;DR:  let's go w/deprecation.

still TL;DR:  if i never have to install or manage R again, i will be a 
happy person!

@HyukjinKwon upgrading R is easy.  getting the right mix of R and all of 
the associated packages working "as expected" is a nightmare.

the biggest problem i foresee is if we upgrade R (and all other packages) 
on the workers, every version of spark will be tested against this...  and 
there will be bugs, test failures, and other time consuming (and obtuse) 
problems to debug.  multiply this by every branch, and you can see the rabbit 
hole you've just entered.

for example, a month ago when i finally had time to dive back in to the 
ubuntu port, after finally figuring out how to install R+friends on ubuntu in 
an identical way to the centos workers, i STILL was finding problems w/lintr 
(see:  https://github.com/apache/spark/pull/22896).

anyways:  i'm more than happy to upgrade R and all the packages to 
something much more recent, but i will definitely appreciate some help in the 
game of test-failure whack-a-mole.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22974: [SPARK-22450][WIP][Core][MLLib][FollowUp] Safely registe...

[zhengruifeng]

Github user zhengruifeng commented on the issue:

https://github.com/apache/spark/pull/22974
  
@srowen Yes, this is the problem. I have to register `Param*` before any 
prediction model, but there are too many anonymous classes  in 
`ParamValidators` and other places, and I have not found an efficient way to 
list those anonymous ones.
So I wonder whether we could continue to register all of them.  Or only 
list simple classes like `MultivariateGaussian `?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/23021
  
adding @holdenk, @ueshin and @icexelloss as well.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/23021
  
adding @icexelloss as well.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/23021
  
> Did you test on python3 as well?

Of course!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23019
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98803/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23019
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23019
  
**[Test build #98803 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98803/testReport)**
 for PR 23019 at commit 
[`628f036`](https://github.com/apache/spark/commit/628f036dba1ae741754021fe83dfc93788ff949f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22954: [SPARK-25981][R] Enables Arrow optimization from ...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/22954#discussion_r233292436
  
--- Diff: R/pkg/R/SQLContext.R ---
@@ -189,19 +238,67 @@ createDataFrame <- function(data, schema = NULL, 
samplingRatio = 1.0,
   x
 }
   }
+  data[] <- lapply(data, cleanCols)
 
-  # drop factors and wrap lists
-  data <- setNames(lapply(data, cleanCols), NULL)
+  args <- list(FUN = list, SIMPLIFY = FALSE, USE.NAMES = FALSE)
+  if (arrowEnabled) {
+shouldUseArrow <- tryCatch({
+  stopifnot(length(data) > 0)
+  dataHead <- head(data, 1)
+  # Currenty Arrow optimization does not support POSIXct and raw 
for now.
+  # Also, it does not support explicit float type set by users. It 
leads to
+  # incorrect conversion. We will fall back to the path without 
Arrow optimization.
+  if (any(sapply(dataHead, function(x) is(x, "POSIXct" {
+stop("Arrow optimization with R DataFrame does not support 
POSIXct type yet.")
+  }
+  if (any(sapply(dataHead, is.raw))) {
+stop("Arrow optimization with R DataFrame does not support raw 
type yet.")
+  }
+  if (inherits(schema, "structType")) {
+if (any(sapply(schema$fields(), function(x) 
x$dataType.toString() == "FloatType"))) {
+  stop("Arrow optimization with R DataFrame does not support 
FloatType type yet.")
--- End diff --

I think it's a bug because it always produces a corrupt value when I try to 
use `number` as explicit float types.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/23012
  
@shaneknapp, do you roughly know how difficult it is (and do you have some 
time shortly) to upgrade R from 3.1 to 3.4? I am asking this because I had some 
difficulties when I tried to manually upgrade from a certain low version to 
another non-latest version.

If it's expected to take a while, let's go deprecation step.
If that's expected to be less difficult, let's go saying unsupporting way.

Does this sound okay to you @felixcheung?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/23012#discussion_r233290797
  
--- Diff: docs/index.md ---
@@ -31,7 +31,8 @@ Spark runs on both Windows and UNIX-like systems (e.g. 
Linux, Mac OS). It's easy
 locally on one machine --- all you need is to have `java` installed on 
your system `PATH`,
 or the `JAVA_HOME` environment variable pointing to a Java installation.
 
-Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, 
Spark {{site.SPARK_VERSION}}
+Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. R prior to version 3.4 
is deprecated as of Spark 3.0.
--- End diff --

At least Python version was noted here before.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/23012
  
Hey shane I don’t think we are saying to test multiple R version at all. 
In fact quite the opposite, just the new(er) version at some point in the 
future.

(We don’t have a better solution for packages though. There’s another 
PR for R arrow package for example)



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21957: [SPARK-24994][SQL] When the data type of the fiel...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/21957#discussion_r233287374
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala 
---
@@ -269,7 +269,8 @@ case class FileSourceScanExec(
   }
 
   @transient
-  private val pushedDownFilters = 
dataFilters.flatMap(DataSourceStrategy.translateFilter)
+  private val pushedDownFilters = dataFilters.flatMap(DataSourceStrategy.
+translateFilter(_, !relation.fileFormat.isInstanceOf[ParquetSource]))
--- End diff --

I don't think we accept changes like this. If this is specific to parquet, 
do it in `ParquetFilters`.

And I still prefer to normalize the filters and remove unnecessary cast, 
before pushing filters down to data sources.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22962: [SPARK-25921][PySpark] Fix barrier task run without Barr...

[xuanyuanking]

Github user xuanyuanking commented on the issue:

https://github.com/apache/spark/pull/22962
  
@HyukjinKwon No problem, I'll give a follow up PR to address all your 
comments and rewrite the UT in to a separate class.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23021
  
**[Test build #98808 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98808/testReport)**
 for PR 23021 at commit 
[`a990770`](https://github.com/apache/spark/commit/a990770ca17e49ac6a04fa5f3e62455af5ef7170).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23021
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5010/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23021
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/23021
  
> Could you add some descriptions to run a single test file or a single 
test case if exists?

Done!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22994: [BUILD] refactor dev/lint-python in to something readabl...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/22994
  
nah, it's not urgent at all.

i also tested this locally by breaking various python scripts and
confirming that it caught errors as expected.

On Tue, Nov 13, 2018 at 5:03 PM Hyukjin Kwon 
wrote:

> I haven't taken a look super closely but the idea looks itself okay. Is it
> urgent? if yes, yup. I don't object to go ahead right away. Otherwise,
> might be good to leave open for few days for review comments ..
>
> Let me leave some cc's for @srowen ,
> @felixcheung , @holdenk
>  ..
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22962: [SPARK-25921][PySpark] Fix barrier task run witho...

[xuanyuanking]

Github user xuanyuanking commented on a diff in the pull request:

https://github.com/apache/spark/pull/22962#discussion_r233283645
  
--- Diff: python/pyspark/taskcontext.py ---
@@ -147,8 +147,8 @@ def __init__(self):
 @classmethod
 def _getOrCreate(cls):
 """Internal function to get or create global BarrierTaskContext."""
-if cls._taskContext is None:
-cls._taskContext = BarrierTaskContext()
+if not isinstance(cls._taskContext, BarrierTaskContext):
+cls._taskContext = object.__new__(cls)
--- End diff --

```
could you add some comments to explain it?
```
@cloud-fan Sorry for the less explain and more comments should be done at 
first, will done in follow up PR.
```
Can we get rid of the rewrite all?
we should remove __init__ too
next time please fully describe what's going on in PR description
```
@HyukjinKwon Sorry for the less explain, all these will be done in next 
follow up PR, and the new UT.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23024: [SPARK-26044][WEBUI]Aggregated Metrics table sort based ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23024
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98800/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23024: [SPARK-26044][WEBUI]Aggregated Metrics table sort based ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23024
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23024: [SPARK-26044][WEBUI]Aggregated Metrics table sort based ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23024
  
**[Test build #98800 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98800/testReport)**
 for PR 23024 at commit 
[`ad5669c`](https://github.com/apache/spark/commit/ad5669c162a5790ab75548c6d47fe4d7b52e2583).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/23021
  
Yup!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22994: [BUILD] refactor dev/lint-python in to something readabl...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/22994
  
I haven't taken a look super closely but the idea looks itself okay. Is it 
urgent? if yes, yup. I don't object to go ahead right away. Otherwise, might be 
good to leave open for few days for review comments .. 

Let me leave some cc's for @srowen, @felixcheung, @holdenk ..


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/23021
  
One thing - I would put “pandas” right after test_ so you get the 
natural
logical grouping with sorting by file name.

On Tue, Nov 13, 2018 at 4:58 PM Hyukjin Kwon 
wrote:

> I am going to push after testing and double checking. The line counts
> would look like this
>
>   54 ./test_utils.py
>  199 ./test_catalog.py
>  503 ./test_grouped_agg_pandas_udf.py
>   45 ./test_group.py
>  320 ./test_session.py
>  153 ./test_readwriter.py
>  806 ./test_scalar_pandas_udf.py
>  216 ./test_pandas_udf.py
>  566 ./test_streaming.py
>   55 ./test_conf.py
>   16 ./__init__.py
>  530 ./test_grouped_map_pandas_udf.py
>  157 ./test_column.py
>  654 ./test_udf.py
>  262 ./test_window_pandas_udf.py
>  278 ./test_functions.py
>  263 ./test_context.py
>  138 ./test_serde.py
>  170 ./test_datasources.py
>  399 ./test_arrow.py
>   96 ./test_appsubmit.py
>  944 ./test_types.py
>  737 ./test_dataframe.py
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23019
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98797/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23019
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/23021
  
I am going to push after testing and double checking. The line counts would 
look like this

```
  54 ./test_utils.py
 199 ./test_catalog.py
 503 ./test_grouped_agg_pandas_udf.py
  45 ./test_group.py
 320 ./test_session.py
 153 ./test_readwriter.py
 806 ./test_scalar_pandas_udf.py
 216 ./test_pandas_udf.py
 566 ./test_streaming.py
  55 ./test_conf.py
  16 ./__init__.py
 530 ./test_grouped_map_pandas_udf.py
 157 ./test_column.py
 654 ./test_udf.py
 262 ./test_window_pandas_udf.py
 278 ./test_functions.py
 263 ./test_context.py
 138 ./test_serde.py
 170 ./test_datasources.py
 399 ./test_arrow.py
  96 ./test_appsubmit.py
 944 ./test_types.py
 737 ./test_dataframe.py
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23019: [SPARK-26025][k8s] Speed up docker image build on dev re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23019
  
**[Test build #98797 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98797/testReport)**
 for PR 23019 at commit 
[`007a788`](https://github.com/apache/spark/commit/007a7883a6fa42d622195fbf323f304edc349af4).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22967
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5009/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22967
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22967
  
**[Test build #98807 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98807/testReport)**
 for PR 22967 at commit 
[`52dc4a1`](https://github.com/apache/spark/commit/52dc4a1d625154fb3baab201f9ff3f979b497602).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23024: [SPARK-26044][WEBUI]Aggregated Metrics table sort based ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23024
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23024: [SPARK-26044][WEBUI]Aggregated Metrics table sort based ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23024
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98798/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22967: [SPARK-25956] Make Scala 2.12 as default Scala version i...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/22967
  
Retest this please.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23024: [SPARK-26044][WEBUI]Aggregated Metrics table sort based ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23024
  
**[Test build #98798 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98798/testReport)**
 for PR 23024 at commit 
[`d6099e3`](https://github.com/apache/spark/commit/d6099e3279ddcdf2576503efe2b6f2abc1b71c00).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] Add InMemoryTable filter benchm...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23027
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] Add InMemoryTable filter benchm...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23027
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5008/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] Add InMemoryTable filter benchm...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23027
  
**[Test build #98806 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98806/testReport)**
 for PR 23027 at commit 
[`d0a2a3f`](https://github.com/apache/spark/commit/d0a2a3f4af492fbf69f7774e03d67d4af39cd5c7).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] Add InMemoryTable filter...

[wangyum]

GitHub user wangyum opened a pull request:

https://github.com/apache/spark/pull/23027

[SPARK-26049][SQL][TEST] Add InMemoryTable filter benchmark to 
FilterPushdownBenchmark

## What changes were proposed in this pull request?

Add InMemoryTable filter benchmark to `FilterPushdownBenchmark`.

## How was this patch tested?

manual tests


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wangyum/spark SPARK-26049

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/23027.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #23027


commit d0a2a3f4af492fbf69f7774e03d67d4af39cd5c7
Author: Yuming Wang 
Date:   2018-11-14T00:35:35Z

Add InMemoryTable filter benchmark




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23009: [SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app withou...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23009
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23009: [SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app withou...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/23009
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98801/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23009: [SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app withou...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/23009
  
**[Test build #98801 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98801/testReport)**
 for PR 23009 at commit 
[`c8424af`](https://github.com/apache/spark/commit/c8424aff80e33f9a3f5a7d19a04442c7dac701a4).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org