[GitHub] spark pull request #22569: [SPARK-25542][Core][Test] Move flaky test in Open...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22569#discussion_r221147850
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/OpenHashSetSuite.scala ---
@@ -255,4 +255,16 @@ class OpenHashSetSuite extends SparkFunSuite with
Matchers {
val set = new OpenHashSet[Long](0)
assert(set.size === 0)
}
+
+ test("support for more than 12M items") {
+val cnt = 1200 // 12M
+val set = new OpenHashSet[Int](cnt)
+for (i <- 0 until cnt) {
+ set.add(i)
+
+ val pos1 = set.addWithoutResize(i) & OpenHashSet.POSITION_MASK
+ val pos2 = set.getPos(i)
+ assert(pos1 == pos2)
+}
--- End diff --
oh, the original test performed the count check to see how many values are
invalid. They are invalid values because the index is wrong due to wrong
position mask in OpenHashSet.
This rewritten test tests directly the index of OpenHashSet.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22569: [SPARK-25542][Core][Test] Move flaky test in Open...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22569#discussion_r221147152
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/OpenHashSetSuite.scala ---
@@ -255,4 +255,16 @@ class OpenHashSetSuite extends SparkFunSuite with
Matchers {
val set = new OpenHashSet[Long](0)
assert(set.size === 0)
}
+
+ test("support for more than 12M items") {
+val cnt = 1200 // 12M
+val set = new OpenHashSet[Int](cnt)
+for (i <- 0 until cnt) {
+ set.add(i)
+
+ val pos1 = set.addWithoutResize(i) & OpenHashSet.POSITION_MASK
+ val pos2 = set.getPos(i)
+ assert(pos1 == pos2)
+}
--- End diff --
The original test performed a check by using `map.iterator.count()`. But,
this is not a strong preference.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22544: [SPARK-25522][SQL] Improve type promotion for input argu...
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22544 @cloud-fan Sounds good Wenchen. Thanks for the detailed analysis. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96731/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22574 **[Test build #96731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96731/testReport)** for PR 22574 at commit [`29a9aa0`](https://github.com/apache/spark/commit/29a9aa0a30bea4d126e8668ade66678bdc2ab5fa). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user sadhen commented on the issue: https://github.com/apache/spark/pull/22577 @cloud-fan see: https://github.com/scala/scala/blob/2.12.x/src/compiler/scala/tools/nsc/typechecker/RefChecks.scala --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user sadhen commented on the issue: https://github.com/apache/spark/pull/22577 Error Message: ``` $ dev/change-scala-version.sh 2.12 â spark git:(branch-2.4) â sbt -Dscala.version=2.12.7 sbt (spark)> project core sbt (core)> clean sbt (core)> compile [error] [warn] /Users/rendong/wdi/spark/core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala:178: Option[String] and String are unrelated: they will most likely never compare equal [error] [warn] app.attempts.filter(_.attemptId == attemptId).headOption [error] [warn] ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22576 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96728/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22576 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22576 **[Test build #96728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96728/testReport)** for PR 22576 at commit [`5fdca38`](https://github.com/apache/spark/commit/5fdca38e21448ccf2f24a68fbadc221c87195581). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22567: [SPARK-25550][WEBUI]Environment Page of Spark Job Histor...
Github user shivusondur commented on the issue: https://github.com/apache/spark/pull/22567 @vanzin Please review this patch. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22566 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22566 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96727/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22566 **[Test build #96727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96727/testReport)** for PR 22566 at commit [`cfa157c`](https://github.com/apache/spark/commit/cfa157c3663a74f40a73d749efd55ecd6dae4ffd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22577 **[Test build #96734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96734/testReport)** for PR 22577 at commit [`1fa5618`](https://github.com/apache/spark/commit/1fa56185d69e5ba9a23d911ccaf844a967880068). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22577 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22577: [CORE][MINOR] Fix obvious error and compiling for...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22577#discussion_r221137927
--- Diff:
core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala
---
@@ -175,7 +175,7 @@ private[v1] class OneApplicationAttemptResource extends
AbstractApplicationResou
def getAttempt(): ApplicationAttemptInfo = {
uiRoot.getApplicationInfo(appId)
.flatMap { app =>
-app.attempts.filter(_.attemptId == attemptId).headOption
--- End diff --
Yea, can you share the error message?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22565: [SPARK-25543][K8S] Print debug message iff execIdsRemove...
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/22565 @dongjoon-hyun Thanks for looking. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22577: [CORE][MINOR] Fix obvious error and compiling for...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22577#discussion_r221136204
--- Diff:
core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala
---
@@ -175,7 +175,7 @@ private[v1] class OneApplicationAttemptResource extends
AbstractApplicationResou
def getAttempt(): ApplicationAttemptInfo = {
uiRoot.getApplicationInfo(appId)
.flatMap { app =>
-app.attempts.filter(_.attemptId == attemptId).headOption
--- End diff --
hmm, just curious, why this doesn't compile in 2.12.7?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22569 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22569 **[Test build #96733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96733/testReport)** for PR 22569 at commit [`22bfa80`](https://github.com/apache/spark/commit/22bfa8036ad43534db2e2d4736a8b3ccc44b92b1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22569 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3557/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22577 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22577 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22569 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user sadhen commented on the issue: https://github.com/apache/spark/pull/22577 Dear Maintainers, please backport it to branch-2.4 if it is accepted. @cloud-fan @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22577 **[Test build #96732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96732/testReport)** for PR 22577 at commit [`1fa5618`](https://github.com/apache/spark/commit/1fa56185d69e5ba9a23d911ccaf844a967880068). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22577: [CORE][MINOR] Fix obvious error and compiling for...
GitHub user sadhen opened a pull request: https://github.com/apache/spark/pull/22577 [CORE][MINOR] Fix obvious error and compiling for Scala 2.12.7 ## What changes were proposed in this pull request? Fix an obvious error. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sadhen/spark minor_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22577.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22577 commit 1fa56185d69e5ba9a23d911ccaf844a967880068 Author: Darcy Shen Date: 2018-09-28T03:33:07Z Fix obvious error and compiling for Scala 2.12.7 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22569 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22569 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96726/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22569 **[Test build #96726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96726/testReport)** for PR 22569 at commit [`22bfa80`](https://github.com/apache/spark/commit/22bfa8036ad43534db2e2d4736a8b3ccc44b92b1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22569: [SPARK-25542][Core][Test] Move flaky test in Open...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22569#discussion_r221131588
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/OpenHashSetSuite.scala ---
@@ -255,4 +255,16 @@ class OpenHashSetSuite extends SparkFunSuite with
Matchers {
val set = new OpenHashSet[Long](0)
assert(set.size === 0)
}
+
+ test("support for more than 12M items") {
+val cnt = 1200 // 12M
+val set = new OpenHashSet[Int](cnt)
+for (i <- 0 until cnt) {
+ set.add(i)
+
+ val pos1 = set.addWithoutResize(i) & OpenHashSet.POSITION_MASK
+ val pos2 = set.getPos(i)
+ assert(pos1 == pos2)
+}
--- End diff --
I see. I think that is not what this wants test and there should be other
tests to make sure it works.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96725/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22574 **[Test build #96725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96725/testReport)** for PR 22574 at commit [`8c76e31`](https://github.com/apache/spark/commit/8c76e315bc0e89d9f049b59c82e97d06c8627025). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22569: [SPARK-25542][Core][Test] Move flaky test in Open...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22569#discussion_r221130737
--- Diff:
core/src/test/scala/org/apache/spark/util/collection/OpenHashSetSuite.scala ---
@@ -255,4 +255,16 @@ class OpenHashSetSuite extends SparkFunSuite with
Matchers {
val set = new OpenHashSet[Long](0)
assert(set.size === 0)
}
+
+ test("support for more than 12M items") {
+val cnt = 1200 // 12M
+val set = new OpenHashSet[Int](cnt)
+for (i <- 0 until cnt) {
+ set.add(i)
+
+ val pos1 = set.addWithoutResize(i) & OpenHashSet.POSITION_MASK
+ val pos2 = set.getPos(i)
+ assert(pos1 == pos2)
+}
--- End diff --
My intention was to verify whether small values (e.g. `0, 1, 2, 3`) are
still valid after several resizings.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22573: [SPARK-25558][SQL] Pushdown predicates for nested...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22573#discussion_r221130276
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -437,53 +436,65 @@ object DataSourceStrategy {
* @return a `Some[Filter]` if the input [[Expression]] is convertible,
otherwise a `None`.
*/
protected[sql] def translateFilter(predicate: Expression):
Option[Filter] = {
+// Recursively try to find an attribute name from the top level that
can be pushed down.
+def attrName(e: Expression): Option[String] = e match {
+ // In Spark and many data sources such as parquet, dots are used as
a column path delimiter;
+ // thus, we don't translate such expressions.
+ case a: Attribute if !a.name.contains(".") =>
+Some(a.name)
--- End diff --
JDBC data source seems having no such restrict. So I worry that this change
can cause some regressions.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3556/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22574 **[Test build #96731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96731/testReport)** for PR 22574 at commit [`29a9aa0`](https://github.com/apache/spark/commit/29a9aa0a30bea4d126e8668ade66678bdc2ab5fa). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96730/testReport)** for PR 22466 at commit [`7d28da0`](https://github.com/apache/spark/commit/7d28da044ca80cd2c06c7d089786bec2657d175f). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96730/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96730/testReport)** for PR 22466 at commit [`7d28da0`](https://github.com/apache/spark/commit/7d28da044ca80cd2c06c7d089786bec2657d175f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22573: [SPARK-25558][SQL] Pushdown predicates for nested...
Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/22573#discussion_r221128544
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -437,53 +436,65 @@ object DataSourceStrategy {
* @return a `Some[Filter]` if the input [[Expression]] is convertible,
otherwise a `None`.
*/
protected[sql] def translateFilter(predicate: Expression):
Option[Filter] = {
+// Recursively try to find an attribute name from the top level that
can be pushed down.
+def attrName(e: Expression): Option[String] = e match {
+ // In Spark and many data sources such as parquet, dots are used as
a column path delimiter;
+ // thus, we don't translate such expressions.
+ case a: Attribute if !a.name.contains(".") =>
+Some(a.name)
--- End diff --
Do we have any data source currently supporting `dot` in the column name
with pushdown? The worst case will be no pushdown for those data sources.
I know ORC doesn't work for now. We can have another followup PR to address
this.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22574: [SPARK-25559][SQL] Just remove the unsupported pr...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22574#discussion_r221127868
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala
---
@@ -488,26 +494,25 @@ private[parquet] class ParquetFilters(
.map(_(nameToParquetField(name).fieldName, value))
case sources.And(lhs, rhs) =>
-// At here, it is not safe to just convert one side if we do not
understand the
-// other side. Here is an example used to explain the reason.
-// Let's say we have NOT(a = 2 AND b in ('1')) and we do not
understand how to
-// convert b in ('1'). If we only convert a = 2, we will end up
with a filter
-// NOT(a = 2), which will generate wrong results.
-// Pushing one side of AND down is only safe to do at the top
level.
-// You can see ParquetRelation's initializeLocalJobFunc method as
an example.
-for {
- lhsFilter <- createFilter(schema, lhs)
- rhsFilter <- createFilter(schema, rhs)
-} yield FilterApi.and(lhsFilter, rhsFilter)
+// If the unsupported predicate is in the top level `And`
condition or in the child
+// `And` condition before hitting `Not` or `Or` condition, it can
be safely removed.
+(createFilterHelper(nameToParquetField, lhs, canRemoveOneSideInAnd
= true),
+ createFilterHelper(nameToParquetField, rhs,
canRemoveOneSideInAnd = true)) match {
--- End diff --
Thank you for pinging me, @dbtsai .
Here, instead of `canRemoveOneSideInAnd = true`, we need to use the
received `canRemoveOneSideInAnd`, don't we? Otherwise, this will cause a bug in
the nested `And` clauses. So, could you review and merge the test case,
https://github.com/dbtsai/spark/pull/5 and fix the PR accordingly?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22504 **[Test build #96729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96729/testReport)** for PR 22504 at commit [`e13d6c6`](https://github.com/apache/spark/commit/e13d6c6744baafa7e79e5d37fc62d1a2fecd6f36). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user bersprockets commented on the issue: https://github.com/apache/spark/pull/22504 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96724/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22574 **[Test build #96724 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96724/testReport)** for PR 22574 at commit [`d49d63b`](https://github.com/apache/spark/commit/d49d63bc40a7752990583f9afbd10c68025510b3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459][SQL] Add viewOriginalText back to CatalogT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22458 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459][SQL] Add viewOriginalText back to CatalogT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22458 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96722/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459][SQL] Add viewOriginalText back to CatalogT...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22458 **[Test build #96722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96722/testReport)** for PR 22458 at commit [`94488ea`](https://github.com/apache/spark/commit/94488eac182921188502f6331340ea6b23781690). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22576 **[Test build #96728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96728/testReport)** for PR 22576 at commit [`5fdca38`](https://github.com/apache/spark/commit/5fdca38e21448ccf2f24a68fbadc221c87195581). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22576 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user RussellSpitzer commented on the issue: https://github.com/apache/spark/pull/22576 @hvanhovell Made a full PR for the change we discussed. Also updated the signature to match the new defined types for the registry and Identifier. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22576 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22576: [SPARK-25560][SQL] Allow FunctionInjection in Spa...
GitHub user RussellSpitzer opened a pull request: https://github.com/apache/spark/pull/22576 [SPARK-25560][SQL] Allow FunctionInjection in SparkExtensions This allows an implementer of Spark Session Extensions to utilize a method "injectFunction" which will add a new function to the default Spark Session Catalogue. ## What changes were proposed in this pull request? Adds a new function to SparkSessionExtensions def injectFunction(functionDescription: FunctionDescription) Where function description is a new type type FunctionDescription = (FunctionIdentifier, FunctionBuilder) The functions are loaded in BaseSessionBuilder when the function registry does not have a parent function registry to get loaded from. ## How was this patch tested? New unit tests are added for the extension in SparkSessionExtensionSuite You can merge this pull request into a Git repository by running: $ git pull https://github.com/RussellSpitzer/spark SPARK-25560 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22576.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22576 commit 5fdca38e21448ccf2f24a68fbadc221c87195581 Author: Russell Spitzer Date: 2018-09-28T01:58:21Z [SPARK-25560][SQL] Allow FunctionInjection in SparkExtensions This allows an implementer of Spark Session Extensions to utilize a method "injectFunction" which will add a new function to the default Spark Session Catalogue. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22575: [SPARK-24630][SS][WIP] Support SQLStreaming in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22575 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96721/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22519: [SPARK-25505][SQL] The output order of grouping columns ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22519 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96723/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22519: [SPARK-25505][SQL] The output order of grouping columns ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22519 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22575: [SPARK-24630][SS][WIP] Support SQLStreaming in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22575 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22575: [SPARK-24630][SS][WIP] Support SQLStreaming in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22575 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22570 For me, this PR is too intrusive to merge at this stage. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21688 **[Test build #96721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96721/testReport)** for PR 21688 at commit [`2bd0fcf`](https://github.com/apache/spark/commit/2bd0fcfa5aa89cfd569def82e525bca7cfb469c2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22519: [SPARK-25505][SQL] The output order of grouping columns ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22519 **[Test build #96723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96723/testReport)** for PR 22519 at commit [`2aee8b1`](https://github.com/apache/spark/commit/2aee8b137e6dca733b6529e6b96ef2773c5cb2d4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22575: [SPARK-24630][SS][WIP] Support SQLStreaming in Sp...
GitHub user stczwd opened a pull request: https://github.com/apache/spark/pull/22575 [SPARK-24630][SS][WIP] Support SQLStreaming in Spark ## What changes were proposed in this pull request? This patch propose new support of SQLStreaming in Spark, Please refer [SPARK-24630](https://issues.apache.org/jira/browse/SPARK-24630) for more details. This patch supports: 1. Support create stream table, which can be used as Source and Sink in SQLStreaming; `create table kafka_sql_test using kafka options( isStreaming = 'true', subscribe = 'topic', kafka.bootstrap.servers = 'localhost:9092')` 2. Add keyword 'STREAM' in sql to support SQLStreaming queries; `select stream * from kafka_sql_test` 3. As for those complex queries, they all can be supported as long as SQL and StructStreaming support. ## How was this patch tested? Some UTs are added to verify sqlstreaming. You can merge this pull request into a Git repository by running: $ git pull https://github.com/stczwd/spark sqlstreaming Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22575.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22575 commit af26ea77157a7ff4e0a2c5eecec64c57f73c425d Author: Jackey Lee Date: 2018-09-28T01:04:17Z Support SQLStreaming in Spark: Add keyword 'STREAM'; Support create stream table --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22557: [SPARK-25535][core] Work around bad error handling in co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22557 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96720/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22557: [SPARK-25535][core] Work around bad error handling in co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22557 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22557: [SPARK-25535][core] Work around bad error handling in co...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22557 **[Test build #96720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96720/testReport)** for PR 22557 at commit [`e6e5382`](https://github.com/apache/spark/commit/e6e538282d408b5a87d7fd81e1c2344e41854f5c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22429 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96718/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22429 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22429 **[Test build #96718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96718/testReport)** for PR 22429 at commit [`bd331c5`](https://github.com/apache/spark/commit/bd331c5f8b2e3a0642cc9f818f400af9cbcf37ef). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker ...
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22570 What do we do next? 1. Change `level` to `warning`and commit all changes. In this case `dev/scalastyle` can pass, but IDEA will have a warning:  2. Remove `EmptyInterpolatedStringChecker` and commit all changes. 3. Just close this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96719/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22379 **[Test build #96719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96719/testReport)** for PR 22379 at commit [`b36d96a`](https://github.com/apache/spark/commit/b36d96af3853701690297e5bd89a37b7209a446e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22458: [SPARK-25459][SQL] Add viewOriginalText back to C...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22458 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459][SQL] Add viewOriginalText back to CatalogT...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22458 Thank you, @zheyuan28 . Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22504 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96717/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22504 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22504 **[Test build #96717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96717/testReport)** for PR 22504 at commit [`e13d6c6`](https://github.com/apache/spark/commit/e13d6c6744baafa7e79e5d37fc62d1a2fecd6f36). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22573: [SPARK-25558][SQL] Pushdown predicates for nested...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22573#discussion_r221116417
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -437,53 +436,65 @@ object DataSourceStrategy {
* @return a `Some[Filter]` if the input [[Expression]] is convertible,
otherwise a `None`.
*/
protected[sql] def translateFilter(predicate: Expression):
Option[Filter] = {
+// Recursively try to find an attribute name from the top level that
can be pushed down.
+def attrName(e: Expression): Option[String] = e match {
+ // In Spark and many data sources such as parquet, dots are used as
a column path delimiter;
+ // thus, we don't translate such expressions.
+ case a: Attribute if !a.name.contains(".") =>
+Some(a.name)
--- End diff --
Will this cause regression for data source supporting dot in column name?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459][SQL] Add viewOriginalText back to CatalogT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22458 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459][SQL] Add viewOriginalText back to CatalogT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22458 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96715/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459][SQL] Add viewOriginalText back to CatalogT...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22458 **[Test build #96715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96715/testReport)** for PR 22458 at commit [`772692d`](https://github.com/apache/spark/commit/772692d2bee67d25e65a7a7e61189651a3a5e30b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22566 **[Test build #96727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96727/testReport)** for PR 22566 at commit [`cfa157c`](https://github.com/apache/spark/commit/cfa157c3663a74f40a73d749efd55ecd6dae4ffd). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22566 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3555/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22566 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22573: [SPARK-25558][SQL] Pushdown predicates for nested fields...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22573 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22573: [SPARK-25558][SQL] Pushdown predicates for nested fields...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22573 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96711/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22473: [SPARK-25449][CORE] Heartbeat shouldn't include accumula...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22473 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96712/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22473: [SPARK-25449][CORE] Heartbeat shouldn't include accumula...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22473 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22573: [SPARK-25558][SQL] Pushdown predicates for nested fields...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22573 **[Test build #96711 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96711/testReport)** for PR 22573 at commit [`d59cb55`](https://github.com/apache/spark/commit/d59cb557ca247a74c101f90394f11916b28f2525). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22473: [SPARK-25449][CORE] Heartbeat shouldn't include accumula...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22473 **[Test build #96712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96712/testReport)** for PR 22473 at commit [`e22ac78`](https://github.com/apache/spark/commit/e22ac78b37230a4dfe2393524d78082fdd4d4252). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22573: [SPARK-25558][SQL] Pushdown predicates for nested fields...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22573 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
