[GitHub] [spark] dongjoon-hyun commented on issue #24756: [SPARK-27896][ML] Fix definition of clustering silhouette coefficient for 1-element clusters
dongjoon-hyun commented on issue #24756: [SPARK-27896][ML] Fix definition of clustering silhouette coefficient for 1-element clusters URL: https://github.com/apache/spark/pull/24756#issuecomment-497915523 I reverted this from branch-2.4. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ajithme commented on issue #24438: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmitted event
ajithme commented on issue #24438: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmitted event URL: https://github.com/apache/spark/pull/24438#issuecomment-497914257 @squito @srowen had some rebase due to upstream conflicts, please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module
AmplabJenkins removed a comment on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module URL: https://github.com/apache/spark/pull/24416#issuecomment-497914021 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module
AmplabJenkins removed a comment on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module URL: https://github.com/apache/spark/pull/24416#issuecomment-497914024 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106044/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497913937 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106046/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module
AmplabJenkins commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module URL: https://github.com/apache/spark/pull/24416#issuecomment-497914024 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106044/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module
SparkQA removed a comment on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module URL: https://github.com/apache/spark/pull/24416#issuecomment-497905575 **[Test build #106044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106044/testReport)** for PR 24416 at commit [`9220e78`](https://github.com/apache/spark/commit/9220e78dd422f66dfd8ec71331e2eec71bc955dc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module
AmplabJenkins commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module URL: https://github.com/apache/spark/pull/24416#issuecomment-497914021 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497913935 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497913937 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106046/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497913935 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
SparkQA removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497907136 **[Test build #106046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106046/testReport)** for PR 24752 at commit [`2e2ffd4`](https://github.com/apache/spark/commit/2e2ffd4dfa51be6844d6afdc1cd279072a31cb22). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module
SparkQA commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module URL: https://github.com/apache/spark/pull/24416#issuecomment-497913845 **[Test build #106044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106044/testReport)** for PR 24416 at commit [`9220e78`](https://github.com/apache/spark/commit/9220e78dd422f66dfd8ec71331e2eec71bc955dc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497913779 **[Test build #106046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106046/testReport)** for PR 24752 at commit [`2e2ffd4`](https://github.com/apache/spark/commit/2e2ffd4dfa51be6844d6afdc1cd279072a31cb22). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class TestPythonUDF(name: String) extends TestUDF ` * ` case class TestScalarPandasUDF(name: String) extends TestUDF ` * ` case class TestScalaUDF(name: String) extends TestUDF ` * `case other => throw new RuntimeException(s\"Unknown UDF class [$` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command
dilipbiswal commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command URL: https://github.com/apache/spark/pull/24759#issuecomment-497913574 cc @gatorsmile @maryannxue This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
gengliangwang commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2 URL: https://github.com/apache/spark/pull/24327#issuecomment-497913250 @dongjoon-hyun I think Spark needs to read the actual physical schema for getting the exact names and data types for pushing down filters. If the names or data types are not matched when performing filter push down, it might cause regression. @rdblue has explained this in https://github.com/apache/spark/pull/21696#discussion_r199979463 . With the current DSV2 design, I think we have to implement Parquet V2 in this way. Suggestions are welcome. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497912921 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497912922 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11300/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497912921 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497912922 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11300/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497913031 **[Test build #106050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106050/testReport)** for PR 24741 at commit [`daffc52`](https://github.com/apache/spark/commit/daffc52dafcb8df7d455b7590a7a73993f3b4733). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
gengliangwang commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2 URL: https://github.com/apache/spark/pull/24327#discussion_r289592328 ## File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetLogRedirector.java ## @@ -25,11 +25,11 @@ // Redirects the JUL logging for parquet-mr versions <= 1.8 to SLF4J logging using // SLF4JBridgeHandler. Parquet-mr versions >= 1.9 use SLF4J directly -final class ParquetLogRedirector implements Serializable { +public final class ParquetLogRedirector implements Serializable { Review comment: @mallman @srowen @wangyum Do you know whether we can remove the Parquet log output redirection in Spark 3.0? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497912002 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106049/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE
ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE URL: https://github.com/apache/spark/pull/24762#discussion_r289592089 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala ## @@ -149,6 +149,16 @@ class HiveUDAFSuite extends QueryTest with TestHiveSingleton with SQLTestUtils { } } } + + test("HiveUDAF with 0 rows throws NPE") { +sql("create table abc(a int)") +checkAnswer(sql("select histogram_numeric(a,2) from abc"), Row.fromSeq(Seq.fill(1)(null))) +sql("insert into abc values (1)") +checkAnswer(sql("select histogram_numeric(a,2) from abc"), + Row.fromSeq(Seq(Row.fromTuple((1.0, 1.0))) :: Nil)) +checkAnswer(sql("select histogram_numeric(a,2) from abc where a=3"), + Row.fromSeq(Seq.fill(1)(null))) Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497911822 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11299/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497912001 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497911821 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE
ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE URL: https://github.com/apache/spark/pull/24762#discussion_r289592084 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala ## @@ -149,6 +149,16 @@ class HiveUDAFSuite extends QueryTest with TestHiveSingleton with SQLTestUtils { } } } + + test("HiveUDAF with 0 rows throws NPE") { +sql("create table abc(a int)") +checkAnswer(sql("select histogram_numeric(a,2) from abc"), Row.fromSeq(Seq.fill(1)(null))) +sql("insert into abc values (1)") +checkAnswer(sql("select histogram_numeric(a,2) from abc"), + Row.fromSeq(Seq(Row.fromTuple((1.0, 1.0))) :: Nil)) Review comment: Used Row(Row(1.0, 1.0) :: Nil) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2
gengliangwang commented on a change in pull request #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2 URL: https://github.com/apache/spark/pull/24327#discussion_r289592328 ## File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetLogRedirector.java ## @@ -25,11 +25,11 @@ // Redirects the JUL logging for parquet-mr versions <= 1.8 to SLF4J logging using // SLF4JBridgeHandler. Parquet-mr versions >= 1.9 use SLF4J directly -final class ParquetLogRedirector implements Serializable { +public final class ParquetLogRedirector implements Serializable { Review comment: @mallman @srowen @wangyum Do you know whether we can remove the Parquet log output redirection here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE
ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE URL: https://github.com/apache/spark/pull/24762#discussion_r289592108 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala ## @@ -149,6 +149,16 @@ class HiveUDAFSuite extends QueryTest with TestHiveSingleton with SQLTestUtils { } } } + + test("HiveUDAF with 0 rows throws NPE") { +sql("create table abc(a int)") +checkAnswer(sql("select histogram_numeric(a,2) from abc"), Row.fromSeq(Seq.fill(1)(null))) Review comment: Neat. Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497911923 **[Test build #106049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106049/testReport)** for PR 24741 at commit [`b6eccd0`](https://github.com/apache/spark/commit/b6eccd008552e4b0de61cce552565ed5e813902f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ajithme commented on issue #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE
ajithme commented on issue #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE URL: https://github.com/apache/spark/pull/24762#issuecomment-497912015 I have updated as per the review comments This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497912002 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106049/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
SparkQA removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497911923 **[Test build #106049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106049/testReport)** for PR 24741 at commit [`b6eccd0`](https://github.com/apache/spark/commit/b6eccd008552e4b0de61cce552565ed5e813902f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE
ajithme commented on a change in pull request #24762: [SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE URL: https://github.com/apache/spark/pull/24762#discussion_r289592112 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala ## @@ -149,6 +149,16 @@ class HiveUDAFSuite extends QueryTest with TestHiveSingleton with SQLTestUtils { } } } + + test("HiveUDAF with 0 rows throws NPE") { +sql("create table abc(a int)") Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497912001 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497911821 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497911999 **[Test build #106049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106049/testReport)** for PR 24741 at commit [`b6eccd0`](https://github.com/apache/spark/commit/b6eccd008552e4b0de61cce552565ed5e813902f). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-497911822 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11299/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.
AmplabJenkins removed a comment on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation. URL: https://github.com/apache/spark/pull/24704#issuecomment-497911376 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.
AmplabJenkins removed a comment on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation. URL: https://github.com/apache/spark/pull/24704#issuecomment-497911378 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106043/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jzhuge commented on issue #24763: [SPARK-27909][SQL] Do not run analysis inside CTE substitution
jzhuge commented on issue #24763: [SPARK-27909][SQL] Do not run analysis inside CTE substitution URL: https://github.com/apache/spark/pull/24763#issuecomment-497911364 LGTM. This PR not only fixes cte.sql failures we encountered while enhancing #24741, it also makes the code much easier to reason about. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.
AmplabJenkins commented on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation. URL: https://github.com/apache/spark/pull/24704#issuecomment-497911376 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.
AmplabJenkins commented on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation. URL: https://github.com/apache/spark/pull/24704#issuecomment-497911378 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106043/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.
SparkQA removed a comment on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation. URL: https://github.com/apache/spark/pull/24704#issuecomment-497904675 **[Test build #106043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106043/testReport)** for PR 24704 at commit [`9bf4ece`](https://github.com/apache/spark/commit/9bf4ece2d9d946a18c87dfbef425e012bbe61197). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation.
SparkQA commented on issue #24704: [SPARK-20286][core] Improve logic for timing out executors in dynamic allocation. URL: https://github.com/apache/spark/pull/24704#issuecomment-497911267 **[Test build #106043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106043/testReport)** for PR 24704 at commit [`9bf4ece`](https://github.com/apache/spark/commit/9bf4ece2d9d946a18c87dfbef425e012bbe61197). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #24747: [SPARK-27772][SQL][TEST] SQLTestUtils Refactoring
HyukjinKwon commented on a change in pull request #24747: [SPARK-27772][SQL][TEST] SQLTestUtils Refactoring URL: https://github.com/apache/spark/pull/24747#discussion_r289590278 ## File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala ## @@ -255,59 +255,65 @@ private[sql] trait SQLTestUtilsBase * Drops temporary view `viewNames` after calling `f`. */ protected def withTempView(viewNames: String*)(f: => Unit): Unit = { -try f finally { - // If the test failed part way, we don't want to mask the failure by failing to remove - // temp views that never got created. - try viewNames.foreach(spark.catalog.dropTempView) catch { -case _: NoSuchTableException => - } -} +tryWithFinally(f)(viewNames.foreach(spark.catalog.dropTempView)) } /** * Drops global temporary view `viewNames` after calling `f`. */ protected def withGlobalTempView(viewNames: String*)(f: => Unit): Unit = { -try f finally { - // If the test failed part way, we don't want to mask the failure by failing to remove - // global temp views that never got created. - try viewNames.foreach(spark.catalog.dropGlobalTempView) catch { -case _: NoSuchTableException => - } -} +tryWithFinally(f)(viewNames.foreach(spark.catalog.dropGlobalTempView)) } /** * Drops table `tableName` after calling `f`. */ protected def withTable(tableNames: String*)(f: => Unit): Unit = { -try f finally { - tableNames.foreach { name => -spark.sql(s"DROP TABLE IF EXISTS $name") - } -} +tryWithFinally(f)(tableNames.foreach { name => + spark.sql(s"DROP TABLE IF EXISTS $name") +}) } /** * Drops view `viewName` after calling `f`. */ protected def withView(viewNames: String*)(f: => Unit): Unit = { -try f finally { +tryWithFinally(f)( viewNames.foreach { name => spark.sql(s"DROP VIEW IF EXISTS $name") } -} +) } /** * Drops cache `cacheName` after calling `f`. */ protected def withCache(cacheNames: String*)(f: => Unit): Unit = { -try f finally { - cacheNames.foreach { cacheName => -try uncacheTable(cacheName) catch { - case _: AnalysisException => +tryWithFinally(f)(cacheNames.foreach(uncacheTable)) + } + + /** + * Executes the given tryBlock and then the given finallyBlock no matter whether tryBlock throws + * an exception. If both tryBlock and finallyBlock throw exceptions, the exception thrown + * from the finallyBlock with be added to the exception thrown from tryBlock as a + * suppress exception. It helps to avoid masking the exception from tryBlock with exception + * from finallyBlock + */ + private def tryWithFinally(tryBlock: => Unit)(finallyBlock: => Unit): Unit = { Review comment: okie. fixing this scope is fine This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24700: [SPARK-27834][SQL][R][PYTHON] Make separate PySpark/SparkR vectorization configurations
HyukjinKwon commented on issue #24700: [SPARK-27834][SQL][R][PYTHON] Make separate PySpark/SparkR vectorization configurations URL: https://github.com/apache/spark/pull/24700#issuecomment-497908426 Is everybody happy with it :-) ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497908386 **[Test build #106048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106048/testReport)** for PR 24752 at commit [`9902113`](https://github.com/apache/spark/commit/9902113fb3be069b5400c85b5fabd9663b6b4bc8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497908271 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11298/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497908270 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497908270 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497908271 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11298/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24761: [SPARK-27905] [SQL] Add higher order function 'forall'
HyukjinKwon commented on issue #24761: [SPARK-27905] [SQL] Add higher order function 'forall' URL: https://github.com/apache/spark/pull/24761#issuecomment-497908298 Do you know if any DBMS has this function? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency
AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497907817 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11297/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency
SparkQA commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497907919 **[Test build #106047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106047/testReport)** for PR 24751 at commit [`addb908`](https://github.com/apache/spark/commit/addb9087b34bfb83aec9c300f473b88a08b670d9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency
AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497907814 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency
AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497907817 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11297/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency
AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497907814 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst
AmplabJenkins removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst URL: https://github.com/apache/spark/pull/24723#issuecomment-497907752 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106039/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst
AmplabJenkins removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst URL: https://github.com/apache/spark/pull/24723#issuecomment-497907750 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst
AmplabJenkins commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst URL: https://github.com/apache/spark/pull/24723#issuecomment-497907750 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst
AmplabJenkins commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst URL: https://github.com/apache/spark/pull/24723#issuecomment-497907752 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106039/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency
wangyum commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2][test-maven] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497907745 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497907662 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497907662 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497907665 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106042/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497907665 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106042/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst
SparkQA removed a comment on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst URL: https://github.com/apache/spark/pull/24723#issuecomment-497894928 **[Test build #106039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106039/testReport)** for PR 24723 at commit [`40c94dd`](https://github.com/apache/spark/commit/40c94dd4884927c7ce34d1e4dae165a7a66b0e7d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
SparkQA removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497898601 **[Test build #106042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106042/testReport)** for PR 24374 at commit [`dcc147e`](https://github.com/apache/spark/commit/dcc147ee3682a2762e20d75c6d86450930fbe0aa). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst
SparkQA commented on issue #24723: [SPARK-27857][SQL] Move ALTER TABLE parsing into Catalyst URL: https://github.com/apache/spark/pull/24723#issuecomment-497907646 **[Test build #106039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106039/testReport)** for PR 24723 at commit [`40c94dd`](https://github.com/apache/spark/commit/40c94dd4884927c7ce34d1e4dae165a7a66b0e7d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
SparkQA commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497907535 **[Test build #106042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106042/testReport)** for PR 24374 at commit [`dcc147e`](https://github.com/apache/spark/commit/dcc147ee3682a2762e20d75c6d86450930fbe0aa). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497907411 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11296/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins removed a comment on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497907410 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497907410 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
AmplabJenkins commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497907411 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11296/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
HyukjinKwon commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497907209 I believe I addressed all comments and it's ready for a review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
SparkQA commented on issue #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files URL: https://github.com/apache/spark/pull/24752#issuecomment-497907136 **[Test build #106046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106046/testReport)** for PR 24752 at commit [`2e2ffd4`](https://github.com/apache/spark/commit/2e2ffd4dfa51be6844d6afdc1cd279072a31cb22). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497907015 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106040/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
AmplabJenkins removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497907012 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497907012 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
AmplabJenkins commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497907015 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106040/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
SparkQA removed a comment on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497897187 **[Test build #106040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106040/testReport)** for PR 24374 at commit [`d15a51d`](https://github.com/apache/spark/commit/d15a51d37e4bfd29230253a88defcdc6d0f2ef26). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency
AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497906915 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency
AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497906916 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106041/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency
AmplabJenkins removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497906915 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency
SparkQA removed a comment on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497897881 **[Test build #106041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106041/testReport)** for PR 24751 at commit [`addb908`](https://github.com/apache/spark/commit/addb9087b34bfb83aec9c300f473b88a08b670d9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency
AmplabJenkins commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497906916 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106041/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling
SparkQA commented on issue #24374: [SPARK-27366][CORE] Support GPU Resources in Spark job scheduling URL: https://github.com/apache/spark/pull/24374#issuecomment-497906944 **[Test build #106040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106040/testReport)** for PR 24374 at commit [`d15a51d`](https://github.com/apache/spark/commit/d15a51d37e4bfd29230253a88defcdc6d0f2ef26). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency
SparkQA commented on issue #24751: [SPARK-27831][SQL][TEST][test-hadoop3.2] Move Hive test jars to maven dependency URL: https://github.com/apache/spark/pull/24751#issuecomment-497906792 **[Test build #106041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106041/testReport)** for PR 24751 at commit [`addb908`](https://github.com/apache/spark/commit/addb9087b34bfb83aec9c300f473b88a08b670d9). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class TestHiveVersion(hiveClient: HiveClient)` * `class TestHiveContext(` * ` case class TestTable(name: String, commands: (() => Unit)*)` * ` protected[hive] implicit class SqlCmd(sql: String) ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
SparkQA commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24068#issuecomment-497906747 **[Test build #106045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106045/testReport)** for PR 24068 at commit [`524a1e1`](https://github.com/apache/spark/commit/524a1e15cdd5b38e33a1c59699659b569c001fcf). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
AmplabJenkins removed a comment on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24068#issuecomment-497906649 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
AmplabJenkins removed a comment on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24068#issuecomment-497906650 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11295/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
AmplabJenkins commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24068#issuecomment-497906650 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11295/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
AmplabJenkins commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24068#issuecomment-497906649 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
cloud-fan commented on issue #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24068#issuecomment-497906575 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #24763: [SPARK-27909][SQL] Do not run analysis inside CTE substitution
cloud-fan commented on issue #24763: [SPARK-27909][SQL] Do not run analysis inside CTE substitution URL: https://github.com/apache/spark/pull/24763#issuecomment-497906518 looks reasonable to me, but not very familiar with this part, cc @gatorsmile @hvanhovell This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
cloud-fan commented on a change in pull request #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL URL: https://github.com/apache/spark/pull/24706#discussion_r289589285 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala ## @@ -0,0 +1,380 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.execution.adaptive + +import java.util +import java.util.concurrent.LinkedBlockingQueue + +import scala.collection.JavaConverters._ +import scala.collection.concurrent.TrieMap +import scala.collection.mutable +import scala.concurrent.ExecutionContext + +import org.apache.spark.SparkException +import org.apache.spark.rdd.RDD +import org.apache.spark.sql.SparkSession +import org.apache.spark.sql.catalyst.InternalRow +import org.apache.spark.sql.catalyst.expressions.Attribute +import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, ReturnAnswer} +import org.apache.spark.sql.catalyst.rules.{Rule, RuleExecutor} +import org.apache.spark.sql.execution._ +import org.apache.spark.sql.execution.exchange._ +import org.apache.spark.sql.execution.ui.SparkListenerSQLAdaptiveExecutionUpdate +import org.apache.spark.sql.internal.SQLConf +import org.apache.spark.util.ThreadUtils + +/** + * A root node to execute the query plan adaptively. It splits the query plan into independent + * stages and executes them in order according to their dependencies. The query stage + * materializes its output at the end. When one stage completes, the data statistics of the + * materialized output will be used to optimize the remainder of the query. + * + * To create query stages, we traverse the query tree bottom up. When we hit an exchange node, + * and if all the child query stages of this exchange node are materialized, we create a new + * query stage for this exchange node. The new stage is then materialized asynchronously once it + * is created. + * + * When one query stage finishes materialization, the rest query is re-optimized and planned based + * on the latest statistics provided by all materialized stages. Then we traverse the query plan + * again and create more stages if possible. After all stages have been materialized, we execute + * the rest of the plan. + */ +case class AdaptiveSparkPlanExec( +initialPlan: SparkPlan, +session: SparkSession, +subqueryMap: Map[Long, ExecSubqueryExpression], +stageCache: TrieMap[SparkPlan, QueryStageExec]) + extends LeafExecNode { + + @transient private val executionId = Option( + session.sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY)).map(_.toLong) + + @transient private val lock = new Object() + + // A list of physical plan rules to be applied before creation of query stages. The physical + // plan should reach a final status of query stages (i.e., no more addition or removal of + // Exchange nodes) after running these rules. + @transient private val queryStagePreparationRules: Seq[Rule[SparkPlan]] = Seq( +PlanAdaptiveSubqueries(subqueryMap), +EnsureRequirements(conf) + ) + + // A list of physical optimizer rules to be applied to a new stage before its execution. These + // optimizations should be stage-independent. + @transient private val queryStageOptimizerRules: Seq[Rule[SparkPlan]] = Seq( +CollapseCodegenStages(conf) + ) + + private var currentStageId = 0 + + @volatile private var currentPhysicalPlan = +applyPhysicalRules(initialPlan, queryStagePreparationRules) + + // The logical plan optimizer for re-optimizing the current logical plan. + private object Optimizer extends RuleExecutor[LogicalPlan] { +// TODO add more optimization rules +override protected def batches: Seq[Batch] = Seq() + } + + /** + * Return type for `createQueryStages` + * @param newPlan the new plan with created query stages. + * @param allChildStagesMaterialized whether all child stages have been materialized. + * @param newStages the newly created query stages, including new reused query stages. + */ + private case class CreateStageResult( +newPlan: SparkPlan, +allChildStagesMaterialized: Boolean, +newStages:
[GitHub] [spark] cloud-fan commented on a change in pull request #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
cloud-fan commented on a change in pull request #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL URL: https://github.com/apache/spark/pull/24706#discussion_r289589186 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStageStrategy.scala ## @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.execution.adaptive + +import org.apache.spark.sql.Strategy +import org.apache.spark.sql.catalyst.expressions.PredicateHelper +import org.apache.spark.sql.catalyst.planning.ExtractEquiJoinKeys +import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan} +import org.apache.spark.sql.execution.SparkPlan +import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec, BroadcastNestedLoopJoinExec, BuildLeft, BuildRight} + +/** + * Strategy for plans containing [[LogicalQueryStage]] nodes: + * 1. Transforms [[LogicalQueryStage]] to its corresponding physical plan that is either being + *executed or has already completed execution. + * 2. Transforms [[Join]] which has one child relation already planned and executed as a Review comment: Then we need to add a note that, this rule must be run before `JoinSelection`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module
SparkQA commented on issue #24416: [SPARK-27521][SQL] Move data source v2 to catalyst module URL: https://github.com/apache/spark/pull/24416#issuecomment-497905575 **[Test build #106044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106044/testReport)** for PR 24416 at commit [`9220e78`](https://github.com/apache/spark/commit/9220e78dd422f66dfd8ec71331e2eec71bc955dc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org