[GitHub] spark issue #22385: [SPARK-25400][CORE] Increase test timeouts
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22385 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3006/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22392: [SPARK-23200] Reset Kubernetes-specific config on Checkp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22392 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22392: [SPARK-23200] Reset Kubernetes-specific config on Checkp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22392 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21618: [SPARK-20408][SQL] Get the glob path in parallel ...
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21618#discussion_r216587584
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala ---
@@ -77,6 +80,51 @@ class SparkHadoopUtilSuite extends SparkFunSuite with
Matchers {
})
}
+ test("test expanding glob path") {
--- End diff --
No problem, I'll fix this.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22371: [SPARK-25386][CORE] Don't need to synchronize the IndexS...
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/22371 OK, thanks everyone for the help. Close it --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22371: [SPARK-25386][CORE] Don't need to synchronize the...
Github user ConeyLiu closed the pull request at: https://github.com/apache/spark/pull/22371 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22385: [SPARK-25400][CORE] Increase test timeouts
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22385 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22392: [SPARK-23200] Reset Kubernetes-specific config on Checkp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22392 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22392: [SPARK-23200] Reset Kubernetes-specific config on...
GitHub user ssaavedra opened a pull request: https://github.com/apache/spark/pull/22392 [SPARK-23200] Reset Kubernetes-specific config on Checkpoint restore Several configuration parameters related to Kubernetes need to be reset, as they are changed with each invokation of spark-submit and thus prevents recovery of Spark Streaming tasks. ## What changes were proposed in this pull request? When using the Kubernetes cluster-manager and spawning a Streaming workload, it is important to reset many spark.kubernetes.* properties that are generated by spark-submit but which would get rewritten when restoring a Checkpoint. This is so, because the spark-submit codepath creates Kubernetes resources, such as a ConfigMap, a Secret and other variables, which have an autogenerated name and the previous one will not resolve anymore. In short, this change enables checkpoint restoration for streaming workloads, and thus enables Spark Streaming workloads in Kubernetes, which were not possible to restore from a checkpoint before if the workload went down. ## How was this patch tested? This patch needs would benefit from testing in different k8s clusters. This is similar to the YARN related code for resetting a Spark Streaming workload, but for the Kubernetes scheduler. This PR removes the initcontainers properties that existed before because they are now removed in master. For a previous discussion, see the non-rebased work at: apache-spark-on-k8s#516 You can merge this pull request into a Git repository by running: $ git pull https://github.com/ssaavedra/spark fix-checkpointing-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22392.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22392 commit f457d023e8e488b89b97f5b3b9936831d7fc9bb6 Author: Santiago Saavedra Date: 2018-09-11T07:58:40Z Reset Kubernetes-specific config on Checkpoint restore Several configuration parameters related to Kubernetes need to be reset, as they are changed with each invokation of spark-submit and thus prevents recovery of Spark Streaming tasks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22357: [SPARK-25363][SQL] Fix schema pruning in where clause by...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22357 Can anyone point me out if there are non addressed comments or problems here? Looks pretty good to me. I think this is rather a bandaid, small and safe fix to get into branch-2.4. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22326: [SPARK-25314][SQL] Fix Python UDF accessing attri...
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r216585000
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
---
@@ -1153,12 +1154,35 @@ class FilterPushdownSuite extends PlanTest {
"x.a".attr === Rand(10) && "y.b".attr === 5))
val correctAnswer =
x.where("x.a".attr === 5).join(y.where("y.a".attr === 5 &&
"y.b".attr === 5),
-condition = Some("x.a".attr === Rand(10)))
+joinType = Cross).where("x.a".attr === Rand(10))
--- End diff --
As the code in canEvaluateWithinJoin, we can get the scope relation :
(CannotEvaluateWithinJoin = nonDeterminstic + Unevaluable) > Unevaluable >
PythonUDF.
So for the safety maybe I just limit the change scope to the smallest
PythonUDF only. Need some advise from you thanks :)
https://github.com/apache/spark/blob/0736e72a66735664b191fc363f54e3c522697dba/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala#L104-L120
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22326: [SPARK-25314][SQL] Fix Python UDF accessing attri...
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22326#discussion_r216583509
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
---
@@ -1153,12 +1154,35 @@ class FilterPushdownSuite extends PlanTest {
"x.a".attr === Rand(10) && "y.b".attr === 5))
val correctAnswer =
x.where("x.a".attr === 5).join(y.where("y.a".attr === 5 &&
"y.b".attr === 5),
-condition = Some("x.a".attr === Rand(10)))
+joinType = Cross).where("x.a".attr === Rand(10))
--- End diff --
Yes, I changed this to let the test passing. The original thought is
nondeterministic expression in join condition is not supported yet, so that's
no big
problem.https://github.com/apache/spark/blob/0736e72a66735664b191fc363f54e3c522697dba/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala#L105
https://github.com/apache/spark/blob/0736e72a66735664b191fc363f54e3c522697dba/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala#L1158-L1159
But now I think I should more carefully about this and just limit the cross
join changes only in PythonUDF case. WDYT? @mgaido91 .Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22367: [SPARK-17916][SPARK-25241][SQL][FOLLOWUP] Fix empty stri...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22367 Usually we merge into master and backport to other branches when it's needed. https://spark.apache.org/contributing.html > 5. Open a pull request against the master branch of apache/spark. (Only in special cases would the PR be opened against other branches.) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22369: [SPARK-25072][DOC] Update migration guide for beh...
Github user xuanyuanking closed the pull request at: https://github.com/apache/spark/pull/22369 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22391: [SPARK-25371][SQL][BACKPORT-2.3] struct() should allow b...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22391 **[Test build #95927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95927/testReport)** for PR 22391 at commit [`66b6bd5`](https://github.com/apache/spark/commit/66b6bd5b7e1538d60915b62fa1155dcd86f3411a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22369: [SPARK-25072][DOC] Update migration guide for behavior c...
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/22369 As the comment in https://github.com/apache/spark/pull/22140#issuecomment-419997180, I think this doc change is no more needed, I just close this, thanks @BryanCutler and @HyukjinKwon ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22391: [SPARK-25371][SQL][BACKPORT-2.3] struct() should allow b...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22391 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22391: [SPARK-25371][SQL][BACKPORT-2.3] struct() should allow b...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22391 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3005/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22378 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22378: [SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STO...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22378#discussion_r216580577
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala ---
@@ -750,4 +751,27 @@ class InsertSuite extends QueryTest with
TestHiveSingleton with BeforeAndAfter
}
}
}
+
+ Seq("LOCAL", "").foreach { local =>
+Seq(true, false).foreach { caseSensitivity =>
+ Seq("orc", "parquet").foreach { format =>
+test(s"SPARK-25389 INSERT OVERWRITE $local DIRECTORY ... STORED AS
with duplicated names" +
+ s"(caseSensitivity=$caseSensitivity, format=$format)") {
+ withTempDir { dir =>
+withSQLConf(SQLConf.CASE_SENSITIVE.key -> s"$caseSensitivity")
{
+ val m = intercept[AnalysisException] {
+sql(
+ s"""
+ |INSERT OVERWRITE $local DIRECTORY '${dir.toURI}'
+ |STORED AS $format
+ |SELECT 'id', 'id2' ${if (caseSensitivity) "id" else
"ID"}
--- End diff --
do we miss a comma after `'id2'`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22391: [SPARK-25371][SQL][BACKPORT-2.3] struct() should allow b...
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/22391 cc @cloud-fan @dongjoon-hyun --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22391: [SPARK-25371][SQL][BACKPORT-2.3] struct() should ...
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/22391 [SPARK-25371][SQL][BACKPORT-2.3] struct() should allow being called with 0 args ## What changes were proposed in this pull request? SPARK-21281 introduced a check for the inputs of `CreateStructLike` to be non-empty. This means that `struct()`, which was previously considered valid, now throws an Exception. This behavior change was introduced in 2.3.0. The change may break users' application on upgrade and it causes `VectorAssembler` to fail when an empty `inputCols` is defined. The PR removes the added check making `struct()` valid again. ## How was this patch tested? added UT You can merge this pull request into a Git repository by running: $ git pull https://github.com/mgaido91/spark SPARK-25371_2.3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22391.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22391 commit 66b6bd5b7e1538d60915b62fa1155dcd86f3411a Author: Marco Gaido Date: 2018-09-11T06:16:56Z [SPARK-25371][SQL] struct() should allow being called with 0 args SPARK-21281 introduced a check for the inputs of `CreateStructLike` to be non-empty. This means that `struct()`, which was previously considered valid, now throws an Exception. This behavior change was introduced in 2.3.0. The change may break users' application on upgrade and it causes `VectorAssembler` to fail when an empty `inputCols` is defined. The PR removes the added check making `struct()` valid again. added UT Closes #22373 from mgaido91/SPARK-25371. Authored-by: Marco Gaido Signed-off-by: Wenchen Fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22371: [SPARK-25386][CORE] Don't need to synchronize the IndexS...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22371 My opinion is, it's not worth to spend time on it. The lock is not likely to be a bottleneck and it's better to keep it simple even it's sub-optimal. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22371: [SPARK-25386][CORE] Don't need to synchronize the IndexS...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22371 @ConeyLiu we may have an executor lost and then come back, and may have 2 same tasks running on the same executor. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22343: [SPARK-25391][SQL] Make behaviors consistent when conver...
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22343
Thank you for the pointer, @seancxmao . And thank you for clarification,
@cloud-fan .
It looks like we are re-creating correctness issue somewhat in this PR when
`caseSensitive=true`.
**BEFORE THIS PR (master)**
```scala
scala> sql("INSERT OVERWRITE DIRECTORY '/tmp/hive_t' STORED AS PARQUET
SELECT 'A', 'a'")
scala> sql("CREATE TABLE hive_t(a STRING) STORED AS PARQUET LOCATION
'/tmp/hive_t'")
scala> sql("CREATE TABLE spark_t(a STRING) USING PARQUET LOCATION
'/tmp/hive_t'")
scala> sql("set spark.sql.caseSensitive=true")
scala> spark.table("hive_t").show
+---+
| a|
+---+
| a|
+---+
scala> spark.table("spark_t").show
+---+
| a|
+---+
| a|
+---+
```
**AFTER THIS PR**
```scala
scala> sql("set spark.sql.caseSensitive=true")
scala> spark.table("hive_t").show
+---+
| a|
+---+
| A|
+---+
scala> spark.table("spark_t").show
+---+
| a|
+---+
| a|
+---+
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22343: [SPARK-25391][SQL] Make behaviors consistent when conver...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22343 To clarify: this is just a workaround when we hit a problematic(having case-insensitive duplicated filed names in the parquet file) hive parquet tables and we want to read it with the native parquet reader. The hive behaivor is weird but we need to follow it as we are reading a hive table. Personally I think it's not a big deal. If the hive table is malformed, I think we don't have to follow hive's bugy behavior. If people are confused by this patch and think this doesn't worth, I'm ok to just leave it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22390: [SPARK-25402][SQL] Null handling in BooleanSimplificatio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22390 **[Test build #95923 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95923/testReport)** for PR 22390 at commit [`cf863bb`](https://github.com/apache/spark/commit/cf863bb1cbccea16c6838950a1daa3070fa3b6e3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22390: [SPARK-25402][SQL] Null handling in BooleanSimplificatio...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22390 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22390: [SPARK-25402][SQL] Null handling in BooleanSimplificatio...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22390 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95923/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22390: [SPARK-25402][SQL] Null handling in BooleanSimpli...
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22390#discussion_r216576135
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
---
@@ -263,10 +263,12 @@ object BooleanSimplification extends
Rule[LogicalPlan] with PredicateHelper {
case TrueLiteral Or _ => TrueLiteral
case _ Or TrueLiteral => TrueLiteral
- case a And b if Not(a).semanticEquals(b) => FalseLiteral
--- End diff --
+1
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22390: [SPARK-25402][SQL] Null handling in BooleanSimpli...
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22390#discussion_r216574955
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala
---
@@ -48,6 +48,14 @@ class BooleanSimplificationSuite extends PlanTest with
PredicateHelper {
testRelation.output, Seq(Row(1, 2, 3, "abc"))
)
+ val testNotnullableRelation = LocalRelation('a.int.notNull,
'b.int.notNull, 'c.int.notNull,
--- End diff --
Nit: `testNotnullableRelation` => `testNotNullableRelation`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22390: [SPARK-25402][SQL] Null handling in BooleanSimpli...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22390#discussion_r216575397
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
---
@@ -263,10 +263,12 @@ object BooleanSimplification extends
Rule[LogicalPlan] with PredicateHelper {
case TrueLiteral Or _ => TrueLiteral
case _ Or TrueLiteral => TrueLiteral
- case a And b if Not(a).semanticEquals(b) => FalseLiteral
--- End diff --
how about
```
case a And b if Not(a).semanticEquals(b) || a.semanticEquals(Not(b)) =>
If(IsNull(a), null, FalseLiteral)
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22390: [SPARK-25402][SQL] Null handling in BooleanSimpli...
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22390#discussion_r216575336
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala
---
@@ -48,6 +48,14 @@ class BooleanSimplificationSuite extends PlanTest with
PredicateHelper {
testRelation.output, Seq(Row(1, 2, 3, "abc"))
)
+ val testNotnullableRelation = LocalRelation('a.int.notNull,
'b.int.notNull, 'c.int.notNull,
+'d.string.notNull, 'e.boolean.notNull, 'f.boolean.notNull,
'g.boolean.notNull,
+'h.boolean.notNull)
+
+ val testNotnullableRelationWithData = LocalRelation.fromExternalRows(
--- End diff --
`testNotnullableRelationWithData` => `testNotNullableRelationWithData`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22341: [SPARK-24889][Core] Update block info when unpersist rdd...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22341 @vanzin Thanks for the good review! I've updated this to address them all. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22341: [SPARK-24889][Core] Update block info when unpersist rdd...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22341 **[Test build #95926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95926/testReport)** for PR 22341 at commit [`7c76790`](https://github.com/apache/spark/commit/7c76790c9fab6d695aa45f7efbba70c0e2a8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22341: [SPARK-24889][Core] Update block info when unpersist rdd...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22341 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22341: [SPARK-24889][Core] Update block info when unpersist rdd...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22341 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3004/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22341: [SPARK-24889][Core] Update block info when unpers...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22341#discussion_r216574255
--- Diff:
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -646,7 +647,47 @@ private[spark] class AppStatusListener(
}
override def onUnpersistRDD(event: SparkListenerUnpersistRDD): Unit = {
-liveRDDs.remove(event.rddId)
+liveRDDs.remove(event.rddId).foreach { liveRDD =>
+ val executorsToUpdate = new HashSet[LiveExecutor]()
--- End diff --
Ok. Good point for me. I updated the change to remove the hash set now.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to su...
Github user wangyum closed the pull request at: https://github.com/apache/spark/pull/20504 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22364: [SPARK-25379][SQL] Improve AttributeSet and Colum...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22364#discussion_r216572034
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeSet.scala
---
@@ -39,10 +41,15 @@ object AttributeSet {
/** Constructs a new [[AttributeSet]] given a sequence of [[Expression
Expressions]]. */
def apply(baseSet: Iterable[Expression]): AttributeSet = {
-new AttributeSet(
- baseSet
-.flatMap(_.references)
-.map(new AttributeEquals(_)).toSet)
--- End diff --
Thank you for your comment and help here!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22374: [SPARK-25387][SQL] Fix for NPE caused by bad CSV ...
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22374#discussion_r216569630
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
---
@@ -1700,4 +1700,13 @@ class CSVSuite extends QueryTest with
SharedSQLContext with SQLTestUtils with Te
checkCount(2)
countForMalformedCSV(0, Seq(""))
}
+
+ test("SPARK-25387: bad input should not cause NPE") {
+val schema = StructType(StructField("a", IntegerType) :: Nil)
+val input = spark.createDataset(Seq("\u\u\u0001234"))
--- End diff --
The `parseLine` method can return `null` in many cases. See:
https://github.com/uniVocity/univocity-parsers/blob/f616d151b48150bc9cb98943f9b6f8353b704359/src/main/java/com/univocity/parsers/common/AbstractParser.java#L663
https://github.com/uniVocity/univocity-parsers/blob/f616d151b48150bc9cb98943f9b6f8353b704359/src/main/java/com/univocity/parsers/common/AbstractParser.java#L678
It is normal way for the method to indicate about an error.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22387: [SPARK-25313][SQL][FOLLOW-UP][BACKPORT-2.3] Fix InsertIn...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22387 **[Test build #95925 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95925/testReport)** for PR 22387 at commit [`a7b857c`](https://github.com/apache/spark/commit/a7b857c69fa20615108413d6f17a87978ca44ae2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22387: [SPARK-25313][SQL][FOLLOW-UP][BACKPORT-2.3] Fix InsertIn...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22387 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3003/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22388: Revert [SPARK-24882][SQL] improve data source v2 API fro...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22388 yes --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22387: [SPARK-25313][SQL][FOLLOW-UP][BACKPORT-2.3] Fix InsertIn...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22387 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22387: [SPARK-25313][SQL][FOLLOW-UP][BACKPORT-2.3] Fix InsertIn...
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22387 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to support a...
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/20504 I will close it now. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22357: [SPARK-25363][SQL] Fix schema pruning in where cl...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22357#discussion_r216565635
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala
---
@@ -110,7 +110,17 @@ private[sql] object ParquetSchemaPruning extends
Rule[LogicalPlan] {
val projectionRootFields = projects.flatMap(getRootFields)
val filterRootFields = filters.flatMap(getRootFields)
-(projectionRootFields ++ filterRootFields).distinct
+// Kind of expressions don't need to access any fields of a root
fields, e.g., `IsNotNull`.
+// For them, if there are any nested fields accessed in the query, we
don't need to add root
+// field access of above expressions.
+// For example, for a query `SELECT name.first FROM contacts WHERE
name IS NOT NULL`,
+// we don't need to read nested fields of `name` struct other than
`first` field.
--- End diff --
> I checked in the ParquetFilter, IsNotNull(employer) will be ignored since
it's not a valid parquet filter as parquet doesn't support pushdown on the
struct; thus, with this PR, this query will return wrong answer.
We may not worry about wrong answer from datasource like Parquet in
predicate pushdown. As not all predicates are supported by pushdown, we always
have a SparkSQL Filter on top of scan node to make sure to receive correct
answer.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22389: [SPARK-17916][SPARK-25241][SQL][FOLLOW-UP] Fix empty str...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22389 **[Test build #95924 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95924/testReport)** for PR 22389 at commit [`9a04d87`](https://github.com/apache/spark/commit/9a04d87fb4fc52490dafe5dcd0dea85c4f1b3c5d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22390: [SPARK-25402][SQL] Null handling in BooleanSimplificatio...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22390 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22390: [SPARK-25402][SQL] Null handling in BooleanSimplificatio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22390 **[Test build #95923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95923/testReport)** for PR 22390 at commit [`cf863bb`](https://github.com/apache/spark/commit/cf863bb1cbccea16c6838950a1daa3070fa3b6e3). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22390: [SPARK-25402][SQL] Null handling in BooleanSimplificatio...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22390 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3002/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22389: [SPARK-17916][SPARK-25241][SQL][FOLLOW-UP] Fix empty str...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22389 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22390: [SPARK-25402][SQL] Null handling in BooleanSimplificatio...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22390 cc @cloud-fan @adrian-ionescu --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22390: [SPARK-25402][SQL] Null handling in BooleanSimpli...
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/22390 [SPARK-25402][SQL] Null handling in BooleanSimplification ## What changes were proposed in this pull request? This PR is to fix the null handling in BooleanSimplification. In the rule BooleanSimplification, there are two cases that do not properly handle null values. The optimization is not right if either side is null. This PR is to fix them. ## How was this patch tested? Added test cases You can merge this pull request into a Git repository by running: $ git pull https://github.com/gatorsmile/spark fixBooleanSimplification Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22390.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22390 commit cf863bb1cbccea16c6838950a1daa3070fa3b6e3 Author: gatorsmile Date: 2018-09-11T07:02:40Z fix. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22376 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22376 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95913/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22192 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95915/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22227: [SPARK-25202] [SQL] Implements split with limit sql func...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/7 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95918/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22387: [SPARK-25313][SQL][FOLLOW-UP][BACKPORT-2.3] Fix InsertIn...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22387 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95921/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22385: [SPARK-25400][CORE] Increase test timeouts
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22385 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95914/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22337: [SPARK-25338][Test] Ensure to call super.beforeAll() and...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22337 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95917/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22227: [SPARK-25202] [SQL] Implements split with limit sql func...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/7 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22192 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22385: [SPARK-25400][CORE] Increase test timeouts
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22385 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22213: [SPARK-25221][DEPLOY] Consistent trailing whitespace tre...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22213 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22385: [SPARK-25400][CORE] Increase test timeouts
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22385 **[Test build #95914 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95914/testReport)** for PR 22385 at commit [`daf76ed`](https://github.com/apache/spark/commit/daf76ed592ed82aa4b390b444c4669ae65c9b355). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22337: [SPARK-25338][Test] Ensure to call super.beforeAll() and...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22337 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21596 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21596 **[Test build #95919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95919/testReport)** for PR 21596 at commit [`bb8b661`](https://github.com/apache/spark/commit/bb8b6619ecf0f92a34ced62d4df4db8204280502). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22389: [SPARK-17916][SPARK-25241][SQL][FOLLOW-UP] Fix empty str...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22389 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95922/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22227: [SPARK-25202] [SQL] Implements split with limit sql func...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/7 **[Test build #95918 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95918/testReport)** for PR 7 at commit [`b12ee88`](https://github.com/apache/spark/commit/b12ee881c1d9025644d075a6124f9e6e465a6378). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22387: [SPARK-25313][SQL][FOLLOW-UP][BACKPORT-2.3] Fix InsertIn...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22387 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21596 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95919/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22387: [SPARK-25313][SQL][FOLLOW-UP][BACKPORT-2.3] Fix InsertIn...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22387 **[Test build #95921 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95921/testReport)** for PR 22387 at commit [`a7b857c`](https://github.com/apache/spark/commit/a7b857c69fa20615108413d6f17a87978ca44ae2). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22213: [SPARK-25221][DEPLOY] Consistent trailing whitespace tre...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22213 **[Test build #95920 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95920/testReport)** for PR 22213 at commit [`76635ce`](https://github.com/apache/spark/commit/76635cef7eab1fceef20fe0f4397460879cb0a43). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22213: [SPARK-25221][DEPLOY] Consistent trailing whitespace tre...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22213 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95920/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22337: [SPARK-25338][Test] Ensure to call super.beforeAll() and...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22337 **[Test build #95917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95917/testReport)** for PR 22337 at commit [`2d9e34a`](https://github.com/apache/spark/commit/2d9e34abdae13efa1fac9e906f331cdd04105e82). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class FlatMapGroupsWithStateSuite extends StateStoreMetricsTest ` * `class StreamingAggregationSuite extends StateStoreMetricsTest with Assertions ` * `class StreamingDeduplicationSuite extends StateStoreMetricsTest ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22389: [SPARK-17916][SPARK-25241][SQL][FOLLOW-UP] Fix empty str...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22389 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22192 **[Test build #95915 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95915/testReport)** for PR 22192 at commit [`447c5e5`](https://github.com/apache/spark/commit/447c5e5974ca2a176026e63518a7a6cf29b78008). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22389: [SPARK-17916][SPARK-25241][SQL][FOLLOW-UP] Fix empty str...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22389 **[Test build #95922 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95922/testReport)** for PR 22389 at commit [`9a04d87`](https://github.com/apache/spark/commit/9a04d87fb4fc52490dafe5dcd0dea85c4f1b3c5d). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22376 **[Test build #95913 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95913/testReport)** for PR 22376 at commit [`4a0cffb`](https://github.com/apache/spark/commit/4a0cffb3ce9e1bede43e6a89fdd7a7b912bf93d2). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22211: [SPARK-23207][SPARK-22905][SPARK-24564][SPARK-25114][SQL...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22211 Hi, @henryr . Since this is merged, could you close this PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13893: [SPARK-14172][SQL] Hive table partition predicate not pa...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13893 Ping, @jiangxb1987 . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22382: [SPARK-23243] [SPARK-20715][CORE][2.2] Fix RDD.repartiti...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22382 Since it's merged, could you close this PR, @bersprockets ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
