[GitHub] [spark] dongjoon-hyun commented on issue #24756: [SPARK-27896][ML] Fix definition of clustering silhouette coefficient for 1-element clusters

2019-05-31 Thread GitBox
dongjoon-hyun commented on issue #24756: [SPARK-27896][ML] Fix definition of 
clustering silhouette coefficient for 1-element clusters
URL: https://github.com/apache/spark/pull/24756#issuecomment-497915523
 
 
   I reverted this from branch-2.4.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] ajithme commented on issue #24438: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmitted event

2019-05-31 Thread GitBox
ajithme commented on issue #24438: [SPARK-23626][CORE] DAGScheduler blocked due 
to JobSubmitted event
URL: https://github.com/apache/spark/pull/24438#issuecomment-497914257
 
 
   @squito @srowen had some rebase due to upstream conflicts, please review


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24416: [SPARK-27521][SQL] Move data 
source v2 to catalyst module
URL: https://github.com/apache/spark/pull/24416#issuecomment-497914021
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24416: [SPARK-27521][SQL] Move data 
source v2 to catalyst module
URL: https://github.com/apache/spark/pull/24416#issuecomment-497914024
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106044/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] 
Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497913937
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106046/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24416: [SPARK-27521][SQL] Move data source v2 
to catalyst module
URL: https://github.com/apache/spark/pull/24416#issuecomment-497914024
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106044/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module

2019-05-31 Thread GitBox
SparkQA removed a comment on issue #24416: [SPARK-27521][SQL] Move data source 
v2 to catalyst module
URL: https://github.com/apache/spark/pull/24416#issuecomment-497905575
 
 
   **[Test build #106044 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106044/testReport)**
 for PR 24416 at commit 
[`9220e78`](https://github.com/apache/spark/commit/9220e78dd422f66dfd8ec71331e2eec71bc955dc).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24416: [SPARK-27521][SQL] Move data source v2 
to catalyst module
URL: https://github.com/apache/spark/pull/24416#issuecomment-497914021
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] 
Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497913935
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an 
integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497913937
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106046/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an 
integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497913935
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
SparkQA removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an 
integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497907136
 
 
   **[Test build #106046 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106046/testReport)**
 for PR 24752 at commit 
[`2e2ffd4`](https://github.com/apache/spark/commit/2e2ffd4dfa51be6844d6afdc1cd279072a31cb22).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module

2019-05-31 Thread GitBox
SparkQA commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to 
catalyst module
URL: https://github.com/apache/spark/pull/24416#issuecomment-497913845
 
 
   **[Test build #106044 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106044/testReport)**
 for PR 24416 at commit 
[`9220e78`](https://github.com/apache/spark/commit/9220e78dd422f66dfd8ec71331e2eec71bc955dc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an 
integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497913779
 
 
   **[Test build #106046 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106046/testReport)**
 for PR 24752 at commit 
[`2e2ffd4`](https://github.com/apache/spark/commit/2e2ffd4dfa51be6844d6afdc1cd279072a31cb22).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
 * `  case class TestPythonUDF(name: String) extends TestUDF `
 * `  case class TestScalarPandasUDF(name: String) extends TestUDF `
 * `  case class TestScalaUDF(name: String) extends TestUDF `
 * `case other => throw new RuntimeException(s\"Unknown UDF class [$`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dilipbiswal commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command

2019-05-31 Thread GitBox
dilipbiswal commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN 
command
URL: https://github.com/apache/spark/pull/24759#issuecomment-497913574
 
 
   cc @gatorsmile @maryannxue 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] gengliangwang commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2

2019-05-31 Thread GitBox
gengliangwang commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to 
File Data Source V2
URL: https://github.com/apache/spark/pull/24327#issuecomment-497913250
 
 
   @dongjoon-hyun I think Spark needs to read the actual physical schema for 
getting the exact names and data types for pushing down filters.  If the names 
or data types are not matched when performing filter push down,  it might cause 
regression.
   @rdblue has explained this in 
https://github.com/apache/spark/pull/21696#discussion_r199979463 .
   
   With the current DSV2 design, I think we have to implement Parquet V2 in 
this way.  Suggestions are welcome.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] 
DataSourceV2: Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497912921
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] 
DataSourceV2: Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497912922
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11300/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: 
Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497912921
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: 
Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497912922
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11300/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from 
multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497913031
 
 
   **[Test build #106050 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106050/testReport)**
 for PR 24741 at commit 
[`daffc52`](https://github.com/apache/spark/commit/daffc52dafcb8df7d455b7590a7a73993f3b4733).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] gengliangwang commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2

2019-05-31 Thread GitBox
gengliangwang commented on a change in pull request #24327: [SPARK-27418][SQL] 
Migrate Parquet to File Data Source V2
URL: https://github.com/apache/spark/pull/24327#discussion_r289592328
 
 

 ##
 File path: 
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetLogRedirector.java
 ##
 @@ -25,11 +25,11 @@
 
 // Redirects the JUL logging for parquet-mr versions <= 1.8 to SLF4J logging 
using
 // SLF4JBridgeHandler. Parquet-mr versions >= 1.9 use SLF4J directly
-final class ParquetLogRedirector implements Serializable {
+public final class ParquetLogRedirector implements Serializable {
 
 Review comment:
   @mallman @srowen @wangyum Do you know whether we can remove the Parquet log 
output redirection in Spark 3.0?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: 
Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497912002
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106049/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE

2019-05-31 Thread GitBox
ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] 
HiveUDAF with 0 rows throw NPE
URL: https://github.com/apache/spark/pull/24762#discussion_r289592089
 
 

 ##
 File path: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala
 ##
 @@ -149,6 +149,16 @@ class HiveUDAFSuite extends QueryTest with 
TestHiveSingleton with SQLTestUtils {
   }
 }
   }
+
+  test("HiveUDAF with 0 rows throws NPE") {
+sql("create table abc(a int)")
+checkAnswer(sql("select histogram_numeric(a,2) from abc"), 
Row.fromSeq(Seq.fill(1)(null)))
+sql("insert into abc values (1)")
+checkAnswer(sql("select histogram_numeric(a,2) from abc"),
+  Row.fromSeq(Seq(Row.fromTuple((1.0, 1.0))) :: Nil))
+checkAnswer(sql("select histogram_numeric(a,2) from abc where a=3"),
+  Row.fromSeq(Seq.fill(1)(null)))
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] 
DataSourceV2: Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497911822
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11299/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: 
Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497912001
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: 
Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497911821
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE

2019-05-31 Thread GitBox
ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] 
HiveUDAF with 0 rows throw NPE
URL: https://github.com/apache/spark/pull/24762#discussion_r289592084
 
 

 ##
 File path: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala
 ##
 @@ -149,6 +149,16 @@ class HiveUDAFSuite extends QueryTest with 
TestHiveSingleton with SQLTestUtils {
   }
 }
   }
+
+  test("HiveUDAF with 0 rows throws NPE") {
+sql("create table abc(a int)")
+checkAnswer(sql("select histogram_numeric(a,2) from abc"), 
Row.fromSeq(Seq.fill(1)(null)))
+sql("insert into abc values (1)")
+checkAnswer(sql("select histogram_numeric(a,2) from abc"),
+  Row.fromSeq(Seq(Row.fromTuple((1.0, 1.0))) :: Nil))
 
 Review comment:
   Used  Row(Row(1.0, 1.0) :: Nil)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] gengliangwang commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2

2019-05-31 Thread GitBox
gengliangwang commented on a change in pull request #24327: [SPARK-27418][SQL] 
Migrate Parquet to File Data Source V2
URL: https://github.com/apache/spark/pull/24327#discussion_r289592328
 
 

 ##
 File path: 
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetLogRedirector.java
 ##
 @@ -25,11 +25,11 @@
 
 // Redirects the JUL logging for parquet-mr versions <= 1.8 to SLF4J logging 
using
 // SLF4JBridgeHandler. Parquet-mr versions >= 1.9 use SLF4J directly
-final class ParquetLogRedirector implements Serializable {
+public final class ParquetLogRedirector implements Serializable {
 
 Review comment:
   @mallman @srowen @wangyum Do you know whether we can remove the Parquet log 
output redirection here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE

2019-05-31 Thread GitBox
ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] 
HiveUDAF with 0 rows throw NPE
URL: https://github.com/apache/spark/pull/24762#discussion_r289592108
 
 

 ##
 File path: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala
 ##
 @@ -149,6 +149,16 @@ class HiveUDAFSuite extends QueryTest with 
TestHiveSingleton with SQLTestUtils {
   }
 }
   }
+
+  test("HiveUDAF with 0 rows throws NPE") {
+sql("create table abc(a int)")
+checkAnswer(sql("select histogram_numeric(a,2) from abc"), 
Row.fromSeq(Seq.fill(1)(null)))
 
 Review comment:
   Neat. Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from 
multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497911923
 
 
   **[Test build #106049 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106049/testReport)**
 for PR 24741 at commit 
[`b6eccd0`](https://github.com/apache/spark/commit/b6eccd008552e4b0de61cce552565ed5e813902f).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] ajithme commented on issue #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE

2019-05-31 Thread GitBox
ajithme commented on issue #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows 
throw NPE
URL: https://github.com/apache/spark/pull/24762#issuecomment-497912015
 
 
   I have updated as per the review comments


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] 
DataSourceV2: Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497912002
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106049/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
SparkQA removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: 
Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497911923
 
 
   **[Test build #106049 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106049/testReport)**
 for PR 24741 at commit 
[`b6eccd0`](https://github.com/apache/spark/commit/b6eccd008552e4b0de61cce552565ed5e813902f).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE

2019-05-31 Thread GitBox
ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] 
HiveUDAF with 0 rows throw NPE
URL: https://github.com/apache/spark/pull/24762#discussion_r289592112
 
 

 ##
 File path: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala
 ##
 @@ -149,6 +149,16 @@ class HiveUDAFSuite extends QueryTest with 
TestHiveSingleton with SQLTestUtils {
   }
 }
   }
+
+  test("HiveUDAF with 0 rows throws NPE") {
+sql("create table abc(a int)")
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] 
DataSourceV2: Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497912001
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] 
DataSourceV2: Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497911821
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from 
multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497911999
 
 
   **[Test build #106049 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106049/testReport)**
 for PR 24741 at commit 
[`b6eccd0`](https://github.com/apache/spark/commit/b6eccd008552e4b0de61cce552565ed5e813902f).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: 
Select from multiple catalogs
URL: https://github.com/apache/spark/pull/24741#issuecomment-497911822
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11299/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24704: [SPARK-20286][core] Improve 
logic for timing out executors in dynamic allocation.
URL: https://github.com/apache/spark/pull/24704#issuecomment-497911376
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24704: [SPARK-20286][core] Improve 
logic for timing out executors in dynamic allocation.
URL: https://github.com/apache/spark/pull/24704#issuecomment-497911378
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106043/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] jzhuge commented on issue #24763: [SPARK-27909][SQL] Do not run analysis inside CTE substitution

2019-05-31 Thread GitBox
jzhuge commented on issue #24763: [SPARK-27909][SQL] Do not run analysis inside 
CTE substitution
URL: https://github.com/apache/spark/pull/24763#issuecomment-497911364
 
 
   LGTM. This PR not only fixes cte.sql failures we encountered while enhancing 
#24741, it also makes the code much easier to reason about.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24704: [SPARK-20286][core] Improve logic for 
timing out executors in dynamic allocation.
URL: https://github.com/apache/spark/pull/24704#issuecomment-497911376
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24704: [SPARK-20286][core] Improve logic for 
timing out executors in dynamic allocation.
URL: https://github.com/apache/spark/pull/24704#issuecomment-497911378
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106043/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.

2019-05-31 Thread GitBox
SparkQA removed a comment on issue #24704: [SPARK-20286][core] Improve logic 
for timing out executors in dynamic allocation.
URL: https://github.com/apache/spark/pull/24704#issuecomment-497904675
 
 
   **[Test build #106043 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106043/testReport)**
 for PR 24704 at commit 
[`9bf4ece`](https://github.com/apache/spark/commit/9bf4ece2d9d946a18c87dfbef425e012bbe61197).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.

2019-05-31 Thread GitBox
SparkQA commented on issue #24704: [SPARK-20286][core] Improve logic for timing 
out executors in dynamic allocation.
URL: https://github.com/apache/spark/pull/24704#issuecomment-497911267
 
 
   **[Test build #106043 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106043/testReport)**
 for PR 24704 at commit 
[`9bf4ece`](https://github.com/apache/spark/commit/9bf4ece2d9d946a18c87dfbef425e012bbe61197).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] HyukjinKwon commented on a change in pull request #24747: [SPARK-27772][SQL][TEST] SQLTestUtils Refactoring

2019-05-31 Thread GitBox
HyukjinKwon commented on a change in pull request #24747: 
[SPARK-27772][SQL][TEST] SQLTestUtils Refactoring
URL: https://github.com/apache/spark/pull/24747#discussion_r289590278
 
 

 ##
 File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala
 ##
 @@ -255,59 +255,65 @@ private[sql] trait SQLTestUtilsBase
* Drops temporary view `viewNames` after calling `f`.
*/
   protected def withTempView(viewNames: String*)(f: => Unit): Unit = {
-try f finally {
-  // If the test failed part way, we don't want to mask the failure by 
failing to remove
-  // temp views that never got created.
-  try viewNames.foreach(spark.catalog.dropTempView) catch {
-case _: NoSuchTableException =>
-  }
-}
+tryWithFinally(f)(viewNames.foreach(spark.catalog.dropTempView))
   }
 
   /**
* Drops global temporary view `viewNames` after calling `f`.
*/
   protected def withGlobalTempView(viewNames: String*)(f: => Unit): Unit = {
-try f finally {
-  // If the test failed part way, we don't want to mask the failure by 
failing to remove
-  // global temp views that never got created.
-  try viewNames.foreach(spark.catalog.dropGlobalTempView) catch {
-case _: NoSuchTableException =>
-  }
-}
+tryWithFinally(f)(viewNames.foreach(spark.catalog.dropGlobalTempView))
   }
 
   /**
* Drops table `tableName` after calling `f`.
*/
   protected def withTable(tableNames: String*)(f: => Unit): Unit = {
-try f finally {
-  tableNames.foreach { name =>
-spark.sql(s"DROP TABLE IF EXISTS $name")
-  }
-}
+tryWithFinally(f)(tableNames.foreach { name =>
+  spark.sql(s"DROP TABLE IF EXISTS $name")
+})
   }
 
   /**
* Drops view `viewName` after calling `f`.
*/
   protected def withView(viewNames: String*)(f: => Unit): Unit = {
-try f finally {
+tryWithFinally(f)(
   viewNames.foreach { name =>
 spark.sql(s"DROP VIEW IF EXISTS $name")
   }
-}
+)
   }
 
   /**
* Drops cache `cacheName` after calling `f`.
*/
   protected def withCache(cacheNames: String*)(f: => Unit): Unit = {
-try f finally {
-  cacheNames.foreach { cacheName =>
-try uncacheTable(cacheName) catch {
-  case _: AnalysisException =>
+tryWithFinally(f)(cacheNames.foreach(uncacheTable))
+  }
+
+  /**
+   * Executes the given tryBlock and then the given finallyBlock no matter 
whether tryBlock throws
+   * an exception. If both tryBlock and finallyBlock throw exceptions, the 
exception thrown
+   * from the finallyBlock with be added to the exception thrown from tryBlock 
as a
+   * suppress exception. It helps to avoid masking the exception from tryBlock 
with exception
+   * from finallyBlock
+   */
+  private def tryWithFinally(tryBlock: => Unit)(finallyBlock: => Unit): Unit = 
{
 
 Review comment:
   okie. fixing this scope is fine


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] HyukjinKwon commented on issue #24700: [SPARK-27834][SQL][R][PYTHON] Make separate PySpark/SparkR vectorization configurations

2019-05-31 Thread GitBox
HyukjinKwon commented on issue #24700: [SPARK-27834][SQL][R][PYTHON] Make 
separate PySpark/SparkR vectorization configurations
URL: https://github.com/apache/spark/pull/24700#issuecomment-497908426
 
 
   Is everybody happy with it :-) ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an 
integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497908386
 
 
   **[Test build #106048 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106048/testReport)**
 for PR 24752 at commit 
[`9902113`](https://github.com/apache/spark/commit/9902113fb3be069b5400c85b5fabd9663b6b4bc8).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] 
Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497908271
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11298/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] 
Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497908270
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an 
integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497908270
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an 
integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497908271
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11298/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] HyukjinKwon commented on issue #24761: [SPARK-27905] [SQL] Add higher order function 'forall'

2019-05-31 Thread GitBox
HyukjinKwon commented on issue #24761: [SPARK-27905] [SQL] Add higher order 
function 'forall'
URL: https://github.com/apache/spark/pull/24761#issuecomment-497908298
 
 
   Do you know if any DBMS has this function?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24751: 
[SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to 
maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497907817
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11297/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
SparkQA commented on issue #24751: 
[SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to 
maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497907919
 
 
   **[Test build #106047 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106047/testReport)**
 for PR 24751 at commit 
[`addb908`](https://github.com/apache/spark/commit/addb9087b34bfb83aec9c300f473b88a08b670d9).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24751: 
[SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to 
maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497907814
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24751: 
[SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to 
maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497907817
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11297/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24751: 
[SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to 
maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497907814
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER 
TABLE parsing into Catalyst
URL: https://github.com/apache/spark/pull/24723#issuecomment-497907752
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106039/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER 
TABLE parsing into Catalyst
URL: https://github.com/apache/spark/pull/24723#issuecomment-497907750
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE 
parsing into Catalyst
URL: https://github.com/apache/spark/pull/24723#issuecomment-497907750
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE 
parsing into Catalyst
URL: https://github.com/apache/spark/pull/24723#issuecomment-497907752
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106039/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
wangyum commented on issue #24751: 
[SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to 
maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497907745
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support 
GPU Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497907662
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU 
Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497907662
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support 
GPU Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497907665
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106042/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU 
Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497907665
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106042/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst

2019-05-31 Thread GitBox
SparkQA removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE 
parsing into Catalyst
URL: https://github.com/apache/spark/pull/24723#issuecomment-497894928
 
 
   **[Test build #106039 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106039/testReport)**
 for PR 24723 at commit 
[`40c94dd`](https://github.com/apache/spark/commit/40c94dd4884927c7ce34d1e4dae165a7a66b0e7d).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
SparkQA removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU 
Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497898601
 
 
   **[Test build #106042 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106042/testReport)**
 for PR 24374 at commit 
[`dcc147e`](https://github.com/apache/spark/commit/dcc147ee3682a2762e20d75c6d86450930fbe0aa).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst

2019-05-31 Thread GitBox
SparkQA commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing 
into Catalyst
URL: https://github.com/apache/spark/pull/24723#issuecomment-497907646
 
 
   **[Test build #106039 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106039/testReport)**
 for PR 24723 at commit 
[`40c94dd`](https://github.com/apache/spark/commit/40c94dd4884927c7ce34d1e4dae165a7a66b0e7d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
SparkQA commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in 
Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497907535
 
 
   **[Test build #106042 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106042/testReport)**
 for PR 24374 at commit 
[`dcc147e`](https://github.com/apache/spark/commit/dcc147ee3682a2762e20d75c6d86450930fbe0aa).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] 
Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497907411
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11296/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] 
Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497907410
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an 
integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497907410
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an 
integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497907411
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11296/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] HyukjinKwon commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
HyukjinKwon commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an 
integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497907209
 
 
   I believe I addressed all comments and it's ready for a review.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

2019-05-31 Thread GitBox
SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an 
integrated test base for Python, Scalar Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#issuecomment-497907136
 
 
   **[Test build #106046 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106046/testReport)**
 for PR 24752 at commit 
[`2e2ffd4`](https://github.com/apache/spark/commit/2e2ffd4dfa51be6844d6afdc1cd279072a31cb22).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support 
GPU Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497907015
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106040/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support 
GPU Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497907012
 
 
   Build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU 
Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497907012
 
 
   Build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU 
Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497907015
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106040/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
SparkQA removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU 
Resources in Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497897187
 
 
   **[Test build #106040 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106040/testReport)**
 for PR 24374 at commit 
[`d15a51d`](https://github.com/apache/spark/commit/d15a51d37e4bfd29230253a88defcdc6d0f2ef26).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24751: 
[SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497906915
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24751: 
[SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497906916
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106041/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24751: 
[SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497906915
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
SparkQA removed a comment on issue #24751: 
[SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497897881
 
 
   **[Test build #106041 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106041/testReport)**
 for PR 24751 at commit 
[`addb908`](https://github.com/apache/spark/commit/addb9087b34bfb83aec9c300f473b88a08b670d9).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24751: 
[SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497906916
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106041/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

2019-05-31 Thread GitBox
SparkQA commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in 
Spark job scheduling
URL: https://github.com/apache/spark/pull/24374#issuecomment-497906944
 
 
   **[Test build #106040 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106040/testReport)**
 for PR 24374 at commit 
[`d15a51d`](https://github.com/apache/spark/commit/d15a51d37e4bfd29230253a88defcdc6d0f2ef26).
* This patch passes all tests.
* This patch **does not merge cleanly**.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency

2019-05-31 Thread GitBox
SparkQA commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] 
Move Hive test jars to maven dependency
URL: https://github.com/apache/spark/pull/24751#issuecomment-497906792
 
 
   **[Test build #106041 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106041/testReport)**
 for PR 24751 at commit 
[`addb908`](https://github.com/apache/spark/commit/addb9087b34bfb83aec9c300f473b88a08b670d9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
 * `case class TestHiveVersion(hiveClient: HiveClient)`
 * `class TestHiveContext(`
 * `  case class TestTable(name: String, commands: (() => Unit)*)`
 * `  protected[hive] implicit class SqlCmd(sql: String) `


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion

2019-05-31 Thread GitBox
SparkQA commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential 
complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#issuecomment-497906747
 
 
   **[Test build #106045 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106045/testReport)**
 for PR 24068 at commit 
[`524a1e1`](https://github.com/apache/spark/commit/524a1e15cdd5b38e33a1c59699659b569c001fcf).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24068: [SPARK-27105][SQL] Optimize 
away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#issuecomment-497906649
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion

2019-05-31 Thread GitBox
AmplabJenkins removed a comment on issue #24068: [SPARK-27105][SQL] Optimize 
away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#issuecomment-497906650
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11295/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24068: [SPARK-27105][SQL] Optimize away 
exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#issuecomment-497906650
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11295/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion

2019-05-31 Thread GitBox
AmplabJenkins commented on issue #24068: [SPARK-27105][SQL] Optimize away 
exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#issuecomment-497906649
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] cloud-fan commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion

2019-05-31 Thread GitBox
cloud-fan commented on issue #24068: [SPARK-27105][SQL] Optimize away 
exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#issuecomment-497906575
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] cloud-fan commented on issue #24763: [SPARK-27909][SQL] Do not run analysis inside CTE substitution

2019-05-31 Thread GitBox
cloud-fan commented on issue #24763: [SPARK-27909][SQL] Do not run analysis 
inside CTE substitution
URL: https://github.com/apache/spark/pull/24763#issuecomment-497906518
 
 
   looks reasonable to me, but not very familiar with this part, cc @gatorsmile 
@hvanhovell 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] cloud-fan commented on a change in pull request #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL

2019-05-31 Thread GitBox
cloud-fan commented on a change in pull request #24706: [SPARK-23128][SQL] A 
new approach to do adaptive execution in Spark SQL
URL: https://github.com/apache/spark/pull/24706#discussion_r289589285
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
 ##
 @@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import java.util
+import java.util.concurrent.LinkedBlockingQueue
+
+import scala.collection.JavaConverters._
+import scala.collection.concurrent.TrieMap
+import scala.collection.mutable
+import scala.concurrent.ExecutionContext
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, ReturnAnswer}
+import org.apache.spark.sql.catalyst.rules.{Rule, RuleExecutor}
+import org.apache.spark.sql.execution._
+import org.apache.spark.sql.execution.exchange._
+import 
org.apache.spark.sql.execution.ui.SparkListenerSQLAdaptiveExecutionUpdate
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * A root node to execute the query plan adaptively. It splits the query plan 
into independent
+ * stages and executes them in order according to their dependencies. The 
query stage
+ * materializes its output at the end. When one stage completes, the data 
statistics of the
+ * materialized output will be used to optimize the remainder of the query.
+ *
+ * To create query stages, we traverse the query tree bottom up. When we hit 
an exchange node,
+ * and if all the child query stages of this exchange node are materialized, 
we create a new
+ * query stage for this exchange node. The new stage is then materialized 
asynchronously once it
+ * is created.
+ *
+ * When one query stage finishes materialization, the rest query is 
re-optimized and planned based
+ * on the latest statistics provided by all materialized stages. Then we 
traverse the query plan
+ * again and create more stages if possible. After all stages have been 
materialized, we execute
+ * the rest of the plan.
+ */
+case class AdaptiveSparkPlanExec(
+initialPlan: SparkPlan,
+session: SparkSession,
+subqueryMap: Map[Long, ExecSubqueryExpression],
+stageCache: TrieMap[SparkPlan, QueryStageExec])
+  extends LeafExecNode {
+
+  @transient private val executionId = Option(
+
session.sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY)).map(_.toLong)
+
+  @transient private val lock = new Object()
+
+  // A list of physical plan rules to be applied before creation of query 
stages. The physical
+  // plan should reach a final status of query stages (i.e., no more addition 
or removal of
+  // Exchange nodes) after running these rules.
+  @transient private val queryStagePreparationRules: Seq[Rule[SparkPlan]] = 
Seq(
+PlanAdaptiveSubqueries(subqueryMap),
+EnsureRequirements(conf)
+  )
+
+  // A list of physical optimizer rules to be applied to a new stage before 
its execution. These
+  // optimizations should be stage-independent.
+  @transient private val queryStageOptimizerRules: Seq[Rule[SparkPlan]] = Seq(
+CollapseCodegenStages(conf)
+  )
+
+  private var currentStageId = 0
+
+  @volatile private var currentPhysicalPlan =
+applyPhysicalRules(initialPlan, queryStagePreparationRules)
+
+  // The logical plan optimizer for re-optimizing the current logical plan.
+  private object Optimizer extends RuleExecutor[LogicalPlan] {
+// TODO add more optimization rules
+override protected def batches: Seq[Batch] = Seq()
+  }
+
+  /**
+   * Return type for `createQueryStages`
+   * @param newPlan the new plan with created query stages.
+   * @param allChildStagesMaterialized whether all child stages have been 
materialized.
+   * @param newStages the newly created query stages, including new reused 
query stages.
+   */
+  private case class CreateStageResult(
+newPlan: SparkPlan,
+allChildStagesMaterialized: Boolean,
+newStages: 

[GitHub] [spark] cloud-fan commented on a change in pull request #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL

2019-05-31 Thread GitBox
cloud-fan commented on a change in pull request #24706: [SPARK-23128][SQL] A 
new approach to do adaptive execution in Spark SQL
URL: https://github.com/apache/spark/pull/24706#discussion_r289589186
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStageStrategy.scala
 ##
 @@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive
+
+import org.apache.spark.sql.Strategy
+import org.apache.spark.sql.catalyst.expressions.PredicateHelper
+import org.apache.spark.sql.catalyst.planning.ExtractEquiJoinKeys
+import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan}
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec, 
BroadcastNestedLoopJoinExec, BuildLeft, BuildRight}
+
+/**
+ * Strategy for plans containing [[LogicalQueryStage]] nodes:
+ * 1. Transforms [[LogicalQueryStage]] to its corresponding physical plan that 
is either being
+ *executed or has already completed execution.
+ * 2. Transforms [[Join]] which has one child relation already planned and 
executed as a
 
 Review comment:
   Then we need to add a note that, this rule must be run before 
`JoinSelection`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module

2019-05-31 Thread GitBox
SparkQA commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to 
catalyst module
URL: https://github.com/apache/spark/pull/24416#issuecomment-497905575
 
 
   **[Test build #106044 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106044/testReport)**
 for PR 24416 at commit 
[`9220e78`](https://github.com/apache/spark/commit/9220e78dd422f66dfd8ec71331e2eec71bc955dc).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



  1   2   3   4   5   6   7   8   9   10   >