[GitHub] [spark] AmplabJenkins commented on issue #24151: [SPARK-26739][SQL][WIP] Standardized Join Types for DataFrames
AmplabJenkins commented on issue #24151: [SPARK-26739][SQL][WIP] Standardized Join Types for DataFrames URL: https://github.com/apache/spark/pull/24151#issuecomment-474695798 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24151: [SPARK-26739][SQL][WIP] Standardized Join Types for DataFrames
AmplabJenkins removed a comment on issue #24151: [SPARK-26739][SQL][WIP] Standardized Join Types for DataFrames URL: https://github.com/apache/spark/pull/24151#issuecomment-474695328 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24151: [SPARK-26739][SQL][WIP] Standardized Join Types for DataFrames
AmplabJenkins commented on issue #24151: [SPARK-26739][SQL][WIP] Standardized Join Types for DataFrames URL: https://github.com/apache/spark/pull/24151#issuecomment-474695437 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24151: [SPARK-26739][SQL][WIP] Standardized Join Types for DataFrames
AmplabJenkins commented on issue #24151: [SPARK-26739][SQL][WIP] Standardized Join Types for DataFrames URL: https://github.com/apache/spark/pull/24151#issuecomment-474695328 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] agrawalpooja opened a new pull request #24151: [SPARK-26739][SQL][WIP] Standardized Join Types for DataFrames
agrawalpooja opened a new pull request #24151: [SPARK-26739][SQL][WIP] Standardized Join Types for DataFrames URL: https://github.com/apache/spark/pull/24151 ## What changes were proposed in this pull request? Tries the address the concern mentioned in [SPARK-26739](https://issues.apache.org/jira/browse/SPARK-26739) To summarise, currently, in the join functions on DataFrames, the join types are defined via a string parameter called joinType. In order for a developer to know which joins are possible, they must look up the API call for join. While this works fine, it can cause the developer to make a typo resulting in improper joins and/or unexpected errors that aren't evident at compile time. The objective of this improvement would be to allow developers to use a common definition for join types (by enum or constants) called JoinTypes. This would contain the possible joins and remove the possibility of a typo. It would also allow Spark to alter the names of the joins in the future without impacting end-users. ## How was this patch tested? Tested via Unit tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
AmplabJenkins removed a comment on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time URL: https://github.com/apache/spark/pull/23951#issuecomment-474693974 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103710/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
AmplabJenkins removed a comment on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time URL: https://github.com/apache/spark/pull/23951#issuecomment-474693972 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
SparkQA removed a comment on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time URL: https://github.com/apache/spark/pull/23951#issuecomment-474691643 **[Test build #103710 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103710/testReport)** for PR 23951 at commit [`e2faee6`](https://github.com/apache/spark/commit/e2faee6b0680cab7a7dd50d809126f4214849601). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
AmplabJenkins commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time URL: https://github.com/apache/spark/pull/23951#issuecomment-474693974 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103710/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
SparkQA commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time URL: https://github.com/apache/spark/pull/23951#issuecomment-474693957 **[Test build #103710 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103710/testReport)** for PR 23951 at commit [`e2faee6`](https://github.com/apache/spark/commit/e2faee6b0680cab7a7dd50d809126f4214849601). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
AmplabJenkins commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time URL: https://github.com/apache/spark/pull/23951#issuecomment-474693972 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
AmplabJenkins removed a comment on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time URL: https://github.com/apache/spark/pull/23951#issuecomment-474692738 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9098/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
AmplabJenkins removed a comment on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time URL: https://github.com/apache/spark/pull/23951#issuecomment-474692736 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
AmplabJenkins commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time URL: https://github.com/apache/spark/pull/23951#issuecomment-474692738 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9098/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
AmplabJenkins commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time URL: https://github.com/apache/spark/pull/23951#issuecomment-474692736 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24119: [SPARK-27182][SQL] Move the conflict source code of the sql/core module to sql/core/v1.2.1
SparkQA commented on issue #24119: [SPARK-27182][SQL] Move the conflict source code of the sql/core module to sql/core/v1.2.1 URL: https://github.com/apache/spark/pull/24119#issuecomment-474691631 **[Test build #103709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103709/testReport)** for PR 24119 at commit [`020d7e7`](https://github.com/apache/spark/commit/020d7e77d84a9af02167800e21fbe4185c8d2e2c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
SparkQA commented on issue #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time URL: https://github.com/apache/spark/pull/23951#issuecomment-474691643 **[Test build #103710 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103710/testReport)** for PR 23951 at commit [`e2faee6`](https://github.com/apache/spark/commit/e2faee6b0680cab7a7dd50d809126f4214849601). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24119: [SPARK-27182][SQL] Move the conflict source code of the sql/core module to sql/core/v1.2.1
AmplabJenkins removed a comment on issue #24119: [SPARK-27182][SQL] Move the conflict source code of the sql/core module to sql/core/v1.2.1 URL: https://github.com/apache/spark/pull/24119#issuecomment-474691293 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24119: [SPARK-27182][SQL] Move the conflict source code of the sql/core module to sql/core/v1.2.1
AmplabJenkins commented on issue #24119: [SPARK-27182][SQL] Move the conflict source code of the sql/core module to sql/core/v1.2.1 URL: https://github.com/apache/spark/pull/24119#issuecomment-474691293 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24119: [SPARK-27182][SQL] Move the conflict source code of the sql/core module to sql/core/v1.2.1
AmplabJenkins removed a comment on issue #24119: [SPARK-27182][SQL] Move the conflict source code of the sql/core module to sql/core/v1.2.1 URL: https://github.com/apache/spark/pull/24119#issuecomment-474691299 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9097/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24119: [SPARK-27182][SQL] Move the conflict source code of the sql/core module to sql/core/v1.2.1
AmplabJenkins commented on issue #24119: [SPARK-27182][SQL] Move the conflict source code of the sql/core module to sql/core/v1.2.1 URL: https://github.com/apache/spark/pull/24119#issuecomment-474691299 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9097/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file.
AmplabJenkins removed a comment on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file. URL: https://github.com/apache/spark/pull/24150#issuecomment-474688295 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file.
AmplabJenkins removed a comment on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file. URL: https://github.com/apache/spark/pull/24150#issuecomment-474688299 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file.
SparkQA commented on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file. URL: https://github.com/apache/spark/pull/24150#issuecomment-474688648 **[Test build #103708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103708/testReport)** for PR 24150 at commit [`edb7440`](https://github.com/apache/spark/commit/edb744015d0fdcec3ce483512887dc078a896c2a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file.
AmplabJenkins commented on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file. URL: https://github.com/apache/spark/pull/24150#issuecomment-474688295 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file.
AmplabJenkins commented on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file. URL: https://github.com/apache/spark/pull/24150#issuecomment-474688299 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal edited a comment on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file.
dilipbiswal edited a comment on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file. URL: https://github.com/apache/spark/pull/24150#issuecomment-474687590 cc @cloud-fan @maropu @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file.
dilipbiswal commented on issue #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file. URL: https://github.com/apache/spark/pull/24150#issuecomment-474687590 cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal opened a new pull request #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file.
dilipbiswal opened a new pull request #24150: [SPARK-27209][SQL] Split parsing of SELECT and INSERT into two top-level rules in the grammar file. URL: https://github.com/apache/spark/pull/24150 ## What changes were proposed in this pull request? Currently in the grammar file the rule `query` is responsible to parse both select and insert statements. As a result, we need to have more semantic checks in the code to guard against in valid insert constructs in a query. Couple of examples are in the `visitCreateView` and `visitAlterView` functions. One other issue is that, we don't catch the `invalid insert constructs` in all the places. Here are couple of examples : ```SQL select * from (insert into bar values (2)); ``` ``` Error in query: unresolved operator 'Project [*]; 'Project [*] +- SubqueryAlias `__auto_generated_subquery_name` +- InsertIntoHiveTable `default`.`bar`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, false, false, [c1] +- Project [cast(col1#18 as int) AS c1#20] +- LocalRelation [col1#18] ``` ```SQL select * from foo where c1 in (insert into bar values (2)) ``` ``` Error in query: cannot resolve '(default.foo.`c1` IN (listquery()))' due to data type mismatch: The number of columns in the left hand side of an IN subquery does not match the number of columns in the output of subquery. #columns in left hand side: 1. #columns in right hand side: 0. Left side columns: [default.foo.`c1`]. Right side columns: [].;; 'Project [*] +- 'Filter c1#6 IN (list#5 []) : +- InsertIntoHiveTable `default`.`bar`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, false, false, [c1] : +- Project [cast(col1#7 as int) AS c1#9] :+- LocalRelation [col1#7] +- SubqueryAlias `default`.`foo` +- HiveTableRelation `default`.`foo`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#6] ``` For both the cases above, we should reject the syntax at parser level. In this PR, we create two top-level parser rules to parse `SELECT` and `INSERT` respectively. ## How was this patch tested? Added tests to PlanParserSuite and removed the semantic check tests from SparkSqlParserSuites. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24145: [SPARK-27201][WebUI] Show full job description on click
SparkQA commented on issue #24145: [SPARK-27201][WebUI] Show full job description on click URL: https://github.com/apache/spark/pull/24145#issuecomment-474683607 **[Test build #103707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103707/testReport)** for PR 24145 at commit [`dafe9f1`](https://github.com/apache/spark/commit/dafe9f1bfc25ba8e8603a606659de8cd558dad6e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24145: [SPARK-27201][WebUI] Show full job description on click
AmplabJenkins removed a comment on issue #24145: [SPARK-27201][WebUI] Show full job description on click URL: https://github.com/apache/spark/pull/24145#issuecomment-474683307 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9094/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer
AmplabJenkins removed a comment on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer URL: https://github.com/apache/spark/pull/24055#issuecomment-474683314 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer
AmplabJenkins removed a comment on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer URL: https://github.com/apache/spark/pull/24055#issuecomment-474683317 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9095/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24145: [SPARK-27201][WebUI] Show full job description on click
AmplabJenkins removed a comment on issue #24145: [SPARK-27201][WebUI] Show full job description on click URL: https://github.com/apache/spark/pull/24145#issuecomment-474683303 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24145: [SPARK-27201][WebUI] Show full job description on click
AmplabJenkins commented on issue #24145: [SPARK-27201][WebUI] Show full job description on click URL: https://github.com/apache/spark/pull/24145#issuecomment-474683307 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9094/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer
AmplabJenkins commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer URL: https://github.com/apache/spark/pull/24055#issuecomment-474683317 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9095/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer
AmplabJenkins commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer URL: https://github.com/apache/spark/pull/24055#issuecomment-474683314 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24145: [SPARK-27201][WebUI] Show full job description on click
AmplabJenkins commented on issue #24145: [SPARK-27201][WebUI] Show full job description on click URL: https://github.com/apache/spark/pull/24145#issuecomment-474683303 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24145: [SPARK-27201][WebUI] Show full job description on click
dongjoon-hyun commented on issue #24145: [SPARK-27201][WebUI] Show full job description on click URL: https://github.com/apache/spark/pull/24145#issuecomment-474682283 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer
SparkQA commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer URL: https://github.com/apache/spark/pull/24055#issuecomment-474682196 **[Test build #103706 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103706/testReport)** for PR 24055 at commit [`82979d8`](https://github.com/apache/spark/commit/82979d8beb3f9a9bebc49d47123e9d35f8d25a2d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer
AmplabJenkins removed a comment on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer URL: https://github.com/apache/spark/pull/24055#issuecomment-474679570 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer
maropu commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer URL: https://github.com/apache/spark/pull/24055#issuecomment-474681979 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #24147: [SPARK-27205][CORE] Remove complicated logic for just leaving warning log when main class is scala.App
HeartSaVioR commented on issue #24147: [SPARK-27205][CORE] Remove complicated logic for just leaving warning log when main class is scala.App URL: https://github.com/apache/spark/pull/24147#issuecomment-474681137 ``` Build was aborted Aborted by anonymous ERROR: Step ?Archive the artifacts? failed: no workspace for SparkPullRequestBuilder #103699 ERROR: Step ?Publish JUnit test result report? failed: no workspace for SparkPullRequestBuilder #103699 ``` Maybe Jenkins messed up again or being rebooted? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats
SparkQA commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats URL: https://github.com/apache/spark/pull/23803#issuecomment-474680941 **[Test build #103704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103704/testReport)** for PR 23803 at commit [`7c96209`](https://github.com/apache/spark/commit/7c96209d8e2ee7aeac822266e6fee91a9f8fa382). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default
SparkQA commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default URL: https://github.com/apache/spark/pull/24143#issuecomment-474680926 **[Test build #103703 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103703/testReport)** for PR 24143 at commit [`5423cd6`](https://github.com/apache/spark/commit/5423cd6b0a724c945c4b2a41815b682ebb09e94a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #19599: [SPARK-22381] [ML] Add StringParam that supports valid options (case insensitive)
SparkQA commented on issue #19599: [SPARK-22381] [ML] Add StringParam that supports valid options (case insensitive) URL: https://github.com/apache/spark/pull/19599#issuecomment-474680960 **[Test build #103705 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103705/testReport)** for PR 19599 at commit [`b79d8db`](https://github.com/apache/spark/commit/b79d8db9406fbd29ef46c8a74f8591d2aace45ee). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default
AmplabJenkins removed a comment on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default URL: https://github.com/apache/spark/pull/24143#issuecomment-474679568 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default
AmplabJenkins removed a comment on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default URL: https://github.com/apache/spark/pull/24143#issuecomment-474680655 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9092/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats
AmplabJenkins removed a comment on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats URL: https://github.com/apache/spark/pull/23803#issuecomment-474680685 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9093/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default
AmplabJenkins removed a comment on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default URL: https://github.com/apache/spark/pull/24143#issuecomment-474680651 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats
AmplabJenkins removed a comment on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats URL: https://github.com/apache/spark/pull/23803#issuecomment-474680684 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats
AmplabJenkins commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats URL: https://github.com/apache/spark/pull/23803#issuecomment-474680685 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9093/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats
AmplabJenkins commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats URL: https://github.com/apache/spark/pull/23803#issuecomment-474680684 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default
AmplabJenkins commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default URL: https://github.com/apache/spark/pull/24143#issuecomment-474680655 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9092/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #24124: [SPARK-27186][SQL] mapPartition to replace map to speedUp Dataset's toLocalIterator process
AngersZh commented on issue #24124: [SPARK-27186][SQL] mapPartition to replace map to speedUp Dataset's toLocalIterator process URL: https://github.com/apache/spark/pull/24124#issuecomment-474680663 > That isn't what happens though. toLocalIterator evaluates partitions one by one. Unless you have more info or benchmarks to verify this, this should be closed. But this method will act before other's call Result one partition bay one partition. Decode action is just dong at the same time when call result ?, not prepare data to wait call result? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default
AmplabJenkins commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default URL: https://github.com/apache/spark/pull/24143#issuecomment-474680651 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats
dongjoon-hyun commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats URL: https://github.com/apache/spark/pull/23803#issuecomment-474680220 BTW, thank you for your first contribution, @venkata91 ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #23912: [SPARK-21029][SS] StreamingQuery should be stopped when the SparkSession is stopped
AmplabJenkins removed a comment on issue #23912: [SPARK-21029][SS] StreamingQuery should be stopped when the SparkSession is stopped URL: https://github.com/apache/spark/pull/23912#issuecomment-474679571 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24044: [WIP][test-hadoop3.2] Test Hadoop 3.2 on jenkins
AmplabJenkins commented on issue #24044: [WIP][test-hadoop3.2] Test Hadoop 3.2 on jenkins URL: https://github.com/apache/spark/pull/24044#issuecomment-474679573 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24147: [SPARK-27205][CORE] Remove complicated logic for just leaving warning log when main class is scala.App
AmplabJenkins commented on issue #24147: [SPARK-27205][CORE] Remove complicated logic for just leaving warning log when main class is scala.App URL: https://github.com/apache/spark/pull/24147#issuecomment-474679569 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23912: [SPARK-21029][SS] StreamingQuery should be stopped when the SparkSession is stopped
AmplabJenkins commented on issue #23912: [SPARK-21029][SS] StreamingQuery should be stopped when the SparkSession is stopped URL: https://github.com/apache/spark/pull/23912#issuecomment-474679572 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23912: [SPARK-21029][SS] StreamingQuery should be stopped when the SparkSession is stopped
AmplabJenkins commented on issue #23912: [SPARK-21029][SS] StreamingQuery should be stopped when the SparkSession is stopped URL: https://github.com/apache/spark/pull/23912#issuecomment-474679571 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer
AmplabJenkins commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer URL: https://github.com/apache/spark/pull/24055#issuecomment-474679570 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default
AmplabJenkins commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default URL: https://github.com/apache/spark/pull/24143#issuecomment-474679568 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] liutang123 commented on a change in pull request #24131: [SPARK-27192][Core] spark.task.cpus should be less or equal than spark.executor.cores
liutang123 commented on a change in pull request #24131: [SPARK-27192][Core]
spark.task.cpus should be less or equal than spark.executor.cores
URL: https://github.com/apache/spark/pull/24131#discussion_r267181362
##
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##
@@ -2679,6 +2688,10 @@ object SparkContext extends Logging {
if (threadCount <= 0) {
throw new SparkException(s"Asked to run locally with $threadCount
threads")
}
+if (cpusPerTask > threadCount) {
+ throw new SparkException(s"$threadCount threads available when run
local[$threads], " +
Review comment:
Please pardon me as my English isn't very good.
I do not understand this comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] liutang123 commented on a change in pull request #24131: [SPARK-27192][Core] spark.task.cpus should be less or equal than spark.executor.cores
liutang123 commented on a change in pull request #24131: [SPARK-27192][Core]
spark.task.cpus should be less or equal than spark.executor.cores
URL: https://github.com/apache/spark/pull/24131#discussion_r267180953
##
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##
@@ -2665,8 +2665,17 @@ object SparkContext extends Logging {
// When running locally, don't try to re-execute tasks on failure.
val MAX_LOCAL_TASK_FAILURES = 1
+val cpusPerTask = sc.conf.get(CPUS_PER_TASK)
+
+def checkClusterExecutorCores(): Unit = if (sc.conf.get(EXECUTOR_CORES) <
cpusPerTask) {
+ throw new SparkException(s"${EXECUTOR_CORES.key} must not be <
${CPUS_PER_TASK.key}.")
+}
+
master match {
case "local" =>
+if (cpusPerTask > 1) {
Review comment:
Because when run local mode, just 1 core is available.
```$SPARK_HOME/bin/spark-shell --master local --conf spark.task.cpus=2
scala>sc.setLogLevel("INFO")
scala>sc.parallelize(1 to 9).collect
```
You can see spark will hang after log `INFO TaskSchedulerImpl: Adding task
set 0.0 with 1 tasks`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] liutang123 commented on a change in pull request #24131: [SPARK-27192][Core] spark.task.cpus should be less or equal than spark.executor.cores
liutang123 commented on a change in pull request #24131: [SPARK-27192][Core]
spark.task.cpus should be less or equal than spark.executor.cores
URL: https://github.com/apache/spark/pull/24131#discussion_r267180953
##
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##
@@ -2665,8 +2665,17 @@ object SparkContext extends Logging {
// When running locally, don't try to re-execute tasks on failure.
val MAX_LOCAL_TASK_FAILURES = 1
+val cpusPerTask = sc.conf.get(CPUS_PER_TASK)
+
+def checkClusterExecutorCores(): Unit = if (sc.conf.get(EXECUTOR_CORES) <
cpusPerTask) {
+ throw new SparkException(s"${EXECUTOR_CORES.key} must not be <
${CPUS_PER_TASK.key}.")
+}
+
master match {
case "local" =>
+if (cpusPerTask > 1) {
Review comment:
Because when run local mode, just 1 core is available.
```$SPARK_HOME/bin/spark-shell --master local --conf spark.task.cpus=2
scala>sc.setLogLevel("INFO")
scala>sc.parallelize(1 to 9).collect```
You can see spark will hang after log `INFO TaskSchedulerImpl: Adding task
set 0.0 with 1 tasks`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
LantaoJin commented on a change in pull request #23951:
[SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
URL: https://github.com/apache/spark/pull/23951#discussion_r267180561
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala
##
@@ -17,24 +17,90 @@
package org.apache.spark.deploy.yarn
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import com.google.common.base.Strings
import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic
+import org.apache.hadoop.net._
+import org.apache.hadoop.util.ReflectionUtils
import org.apache.hadoop.yarn.util.RackResolver
import org.apache.log4j.{Level, Logger}
+import org.apache.spark.internal.Logging
+
/**
- * Wrapper around YARN's [[RackResolver]]. This allows Spark tests to easily
override the
+ * Re-implement YARN's [[RackResolver]]. This allows Spark tests to easily
override the
* default behavior, since YARN's class self-initializes the first time it's
called, and
* future calls all use the initial configuration.
*/
-private[yarn] class SparkRackResolver {
+private[spark] class SparkRackResolver {
// RackResolver logs an INFO message whenever it resolves a rack, which is
way too often.
if (Logger.getLogger(classOf[RackResolver]).getLevel == null) {
Logger.getLogger(classOf[RackResolver]).setLevel(Level.WARN)
}
def resolve(conf: Configuration, hostName: String): String = {
-RackResolver.resolve(conf, hostName).getNetworkLocation()
+SparkRackResolver.coreResolve(conf, List(hostName)).head.getNetworkLocation
+ }
+
+ /**
+ * Added in SPARK-27038.
+ * This should be changed to `RackResolver.resolve(conf, hostNames)`
+ * in hadoop releases with YARN-9332.
+ */
+ def resolve(conf: Configuration, hostNames: List[String]): List[Node] = {
+SparkRackResolver.coreResolve(conf, hostNames)
}
+}
+/**
+ * Utility to resolve the rack for hosts in an efficient manner.
+ * It will cache the rack for individual hosts to avoid
+ * repeatedly performing the same expensive lookup.
+ *
+ * Its logic refers [[org.apache.hadoop.yarn.util.RackResolver]] and enhanced.
+ * This will be unnecessary in hadoop releases with YARN-9332.
+ * With that, we could just directly use
[[org.apache.hadoop.yarn.util.RackResolver]].
+ * In the meantime, this is a re-implementation for spark's use.
+ */
+object SparkRackResolver extends Logging {
+ private var dnsToSwitchMapping: DNSToSwitchMapping = _
+ private var initCalled = false
+
+ def coreResolve(conf: Configuration, hostNames: List[String]): List[Node] = {
+if (!initCalled) {
Review comment:
Let me fix the thread-safe problem first.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24029: [SPARK-27108][SQL] Add parsed SQL plans for create, CTAS.
cloud-fan commented on a change in pull request #24029: [SPARK-27108][SQL] Add
parsed SQL plans for create, CTAS.
URL: https://github.com/apache/spark/pull/24029#discussion_r267180424
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
##
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.net.URI
+
+import org.apache.spark.sql.{AnalysisException, SaveMode}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.AnalysisTest
+import org.apache.spark.sql.catalyst.catalog.{BucketSpec,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{CreateTable,
DataSourceResolution}
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
+
+class PlanResolutionSuite extends AnalysisTest {
+ import CatalystSqlParser._
+
+ def parseAndResolve(query: String): LogicalPlan = {
+DataSourceResolution(conf).apply(parsePlan(query))
+ }
+
+ private def extractTableDesc(sql: String): (CatalogTable, Boolean) = {
+parseAndResolve(sql).collect {
+ case CreateTable(tableDesc, mode, _) => (tableDesc, mode ==
SaveMode.Ignore)
+}.head
+ }
+
+ test("create table - with partitioned by") {
Review comment:
shall we merge these 2 test suites? Or we need to add document to say that,
when adding a new DDL command, we need to add a test in both `DDLParserSuite`
and `PlanResolutionSuite` for different purpose.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24029: [SPARK-27108][SQL] Add parsed SQL plans for create, CTAS.
cloud-fan commented on a change in pull request #24029: [SPARK-27108][SQL] Add
parsed SQL plans for create, CTAS.
URL: https://github.com/apache/spark/pull/24029#discussion_r267180426
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
##
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.net.URI
+
+import org.apache.spark.sql.{AnalysisException, SaveMode}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.AnalysisTest
+import org.apache.spark.sql.catalyst.catalog.{BucketSpec,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{CreateTable,
DataSourceResolution}
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
+
+class PlanResolutionSuite extends AnalysisTest {
+ import CatalystSqlParser._
+
+ def parseAndResolve(query: String): LogicalPlan = {
+DataSourceResolution(conf).apply(parsePlan(query))
+ }
+
+ private def extractTableDesc(sql: String): (CatalogTable, Boolean) = {
+parseAndResolve(sql).collect {
+ case CreateTable(tableDesc, mode, _) => (tableDesc, mode ==
SaveMode.Ignore)
+}.head
+ }
+
+ test("create table - with partitioned by") {
Review comment:
shall we merge these 2 test suites? Or we need to add document to say that,
when adding a new DDL command, we need to add a test in both `DDLParserSuite`
and `PlanResolutionSuite` for different purpose.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24029: [SPARK-27108][SQL] Add parsed SQL plans for create, CTAS.
cloud-fan commented on a change in pull request #24029: [SPARK-27108][SQL] Add
parsed SQL plans for create, CTAS.
URL: https://github.com/apache/spark/pull/24029#discussion_r267180424
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
##
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.net.URI
+
+import org.apache.spark.sql.{AnalysisException, SaveMode}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.AnalysisTest
+import org.apache.spark.sql.catalyst.catalog.{BucketSpec,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{CreateTable,
DataSourceResolution}
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
+
+class PlanResolutionSuite extends AnalysisTest {
+ import CatalystSqlParser._
+
+ def parseAndResolve(query: String): LogicalPlan = {
+DataSourceResolution(conf).apply(parsePlan(query))
+ }
+
+ private def extractTableDesc(sql: String): (CatalogTable, Boolean) = {
+parseAndResolve(sql).collect {
+ case CreateTable(tableDesc, mode, _) => (tableDesc, mode ==
SaveMode.Ignore)
+}.head
+ }
+
+ test("create table - with partitioned by") {
Review comment:
shall we merge these 2 test suites? Or we need to add document to say that,
when adding a new DDL command, we need to add a test in both `DDLParserSuite`
and `PlanResolutionSuite` for different purpose.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24029: [SPARK-27108][SQL] Add parsed SQL plans for create, CTAS.
cloud-fan commented on a change in pull request #24029: [SPARK-27108][SQL] Add
parsed SQL plans for create, CTAS.
URL: https://github.com/apache/spark/pull/24029#discussion_r267180426
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
##
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import java.net.URI
+
+import org.apache.spark.sql.{AnalysisException, SaveMode}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.AnalysisTest
+import org.apache.spark.sql.catalyst.catalog.{BucketSpec,
CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.{CreateTable,
DataSourceResolution}
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
+
+class PlanResolutionSuite extends AnalysisTest {
+ import CatalystSqlParser._
+
+ def parseAndResolve(query: String): LogicalPlan = {
+DataSourceResolution(conf).apply(parsePlan(query))
+ }
+
+ private def extractTableDesc(sql: String): (CatalogTable, Boolean) = {
+parseAndResolve(sql).collect {
+ case CreateTable(tableDesc, mode, _) => (tableDesc, mode ==
SaveMode.Ignore)
+}.head
+ }
+
+ test("create table - with partitioned by") {
Review comment:
shall we merge these 2 test suites? Or we need to add document to say that,
when adding a new DDL command, we need to add a test in both `DDLParserSuite`
and `PlanResolutionSuite` for different purpose.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
LantaoJin commented on a change in pull request #23951:
[SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
URL: https://github.com/apache/spark/pull/23951#discussion_r267180128
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala
##
@@ -17,24 +17,93 @@
package org.apache.spark.deploy.yarn
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import com.google.common.base.Strings
import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic
+import org.apache.hadoop.net._
+import org.apache.hadoop.util.ReflectionUtils
import org.apache.hadoop.yarn.util.RackResolver
import org.apache.log4j.{Level, Logger}
+import org.apache.spark.internal.Logging
+
/**
- * Wrapper around YARN's [[RackResolver]]. This allows Spark tests to easily
override the
+ * Re-implement YARN's [[RackResolver]]. This allows Spark tests to easily
override the
* default behavior, since YARN's class self-initializes the first time it's
called, and
* future calls all use the initial configuration.
*/
-private[yarn] class SparkRackResolver {
+private[spark] class SparkRackResolver {
// RackResolver logs an INFO message whenever it resolves a rack, which is
way too often.
if (Logger.getLogger(classOf[RackResolver]).getLevel == null) {
Logger.getLogger(classOf[RackResolver]).setLevel(Level.WARN)
}
def resolve(conf: Configuration, hostName: String): String = {
-RackResolver.resolve(conf, hostName).getNetworkLocation()
+SparkRackResolver.coreResolve(conf, Seq(hostName)).head.getNetworkLocation
+ }
+
+ /**
+ * Added in SPARK-27038.
+ * This should be changed to `RackResolver.resolve(conf, hostNames)`
+ * in hadoop releases with YARN-9332.
+ */
+ def resolve(conf: Configuration, hostNames: Seq[String]): Seq[Node] = {
+SparkRackResolver.coreResolve(conf, hostNames)
}
+}
+/**
+ * Utility to resolve the rack for hosts in an efficient manner.
+ * It will cache the rack for individual hosts to avoid
+ * repeatedly performing the same expensive lookup.
+ *
+ * Its logic refers [[org.apache.hadoop.yarn.util.RackResolver]] and enhanced.
+ * This will be unnecessary in hadoop releases with YARN-9332.
+ * With that, we could just directly use
[[org.apache.hadoop.yarn.util.RackResolver]].
+ * In the meantime, this is a re-implementation for spark's use.
+ */
+object SparkRackResolver extends Logging {
+ private var dnsToSwitchMapping: DNSToSwitchMapping = _
+ @volatile private var initCalled = false
+
+ def coreResolve(conf: Configuration, hostNames: Seq[String]): Seq[Node] = {
+if (!initCalled) {
+ synchronized {
Review comment:
Let me see... Yes, I will rewrite to double-checked locking.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
LantaoJin commented on a change in pull request #23951:
[SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
URL: https://github.com/apache/spark/pull/23951#discussion_r267179916
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala
##
@@ -17,24 +17,90 @@
package org.apache.spark.deploy.yarn
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import com.google.common.base.Strings
import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic
+import org.apache.hadoop.net._
+import org.apache.hadoop.util.ReflectionUtils
import org.apache.hadoop.yarn.util.RackResolver
import org.apache.log4j.{Level, Logger}
+import org.apache.spark.internal.Logging
+
/**
- * Wrapper around YARN's [[RackResolver]]. This allows Spark tests to easily
override the
+ * Re-implement YARN's [[RackResolver]]. This allows Spark tests to easily
override the
* default behavior, since YARN's class self-initializes the first time it's
called, and
* future calls all use the initial configuration.
*/
-private[yarn] class SparkRackResolver {
+private[spark] class SparkRackResolver {
// RackResolver logs an INFO message whenever it resolves a rack, which is
way too often.
if (Logger.getLogger(classOf[RackResolver]).getLevel == null) {
Logger.getLogger(classOf[RackResolver]).setLevel(Level.WARN)
}
def resolve(conf: Configuration, hostName: String): String = {
-RackResolver.resolve(conf, hostName).getNetworkLocation()
+SparkRackResolver.coreResolve(conf, List(hostName)).head.getNetworkLocation
+ }
+
+ /**
+ * Added in SPARK-27038.
+ * This should be changed to `RackResolver.resolve(conf, hostNames)`
+ * in hadoop releases with YARN-9332.
+ */
+ def resolve(conf: Configuration, hostNames: List[String]): List[Node] = {
+SparkRackResolver.coreResolve(conf, hostNames)
}
+}
+/**
+ * Utility to resolve the rack for hosts in an efficient manner.
+ * It will cache the rack for individual hosts to avoid
+ * repeatedly performing the same expensive lookup.
+ *
+ * Its logic refers [[org.apache.hadoop.yarn.util.RackResolver]] and enhanced.
+ * This will be unnecessary in hadoop releases with YARN-9332.
+ * With that, we could just directly use
[[org.apache.hadoop.yarn.util.RackResolver]].
+ * In the meantime, this is a re-implementation for spark's use.
+ */
+object SparkRackResolver extends Logging {
+ private var dnsToSwitchMapping: DNSToSwitchMapping = _
+ private var initCalled = false
+
+ def coreResolve(conf: Configuration, hostNames: List[String]): List[Node] = {
+if (!initCalled) {
Review comment:
Meanwhile, since `var instance: SparkResolver` is static, how to maintain
more than one SparkResolvers for different `conf`? I am not sure the value of
generating a new resolver for different config.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default
SparkQA removed a comment on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default URL: https://github.com/apache/spark/pull/24143#issuecomment-474623623 **[Test build #103697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103697/testReport)** for PR 24143 at commit [`5423cd6`](https://github.com/apache/spark/commit/5423cd6b0a724c945c4b2a41815b682ebb09e94a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default
SparkQA commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default URL: https://github.com/apache/spark/pull/24143#issuecomment-474677421 **[Test build #103697 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103697/testReport)** for PR 24143 at commit [`5423cd6`](https://github.com/apache/spark/commit/5423cd6b0a724c945c4b2a41815b682ebb09e94a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
LantaoJin commented on a change in pull request #23951:
[SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
URL: https://github.com/apache/spark/pull/23951#discussion_r267179916
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala
##
@@ -17,24 +17,90 @@
package org.apache.spark.deploy.yarn
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import com.google.common.base.Strings
import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic
+import org.apache.hadoop.net._
+import org.apache.hadoop.util.ReflectionUtils
import org.apache.hadoop.yarn.util.RackResolver
import org.apache.log4j.{Level, Logger}
+import org.apache.spark.internal.Logging
+
/**
- * Wrapper around YARN's [[RackResolver]]. This allows Spark tests to easily
override the
+ * Re-implement YARN's [[RackResolver]]. This allows Spark tests to easily
override the
* default behavior, since YARN's class self-initializes the first time it's
called, and
* future calls all use the initial configuration.
*/
-private[yarn] class SparkRackResolver {
+private[spark] class SparkRackResolver {
// RackResolver logs an INFO message whenever it resolves a rack, which is
way too often.
if (Logger.getLogger(classOf[RackResolver]).getLevel == null) {
Logger.getLogger(classOf[RackResolver]).setLevel(Level.WARN)
}
def resolve(conf: Configuration, hostName: String): String = {
-RackResolver.resolve(conf, hostName).getNetworkLocation()
+SparkRackResolver.coreResolve(conf, List(hostName)).head.getNetworkLocation
+ }
+
+ /**
+ * Added in SPARK-27038.
+ * This should be changed to `RackResolver.resolve(conf, hostNames)`
+ * in hadoop releases with YARN-9332.
+ */
+ def resolve(conf: Configuration, hostNames: List[String]): List[Node] = {
+SparkRackResolver.coreResolve(conf, hostNames)
}
+}
+/**
+ * Utility to resolve the rack for hosts in an efficient manner.
+ * It will cache the rack for individual hosts to avoid
+ * repeatedly performing the same expensive lookup.
+ *
+ * Its logic refers [[org.apache.hadoop.yarn.util.RackResolver]] and enhanced.
+ * This will be unnecessary in hadoop releases with YARN-9332.
+ * With that, we could just directly use
[[org.apache.hadoop.yarn.util.RackResolver]].
+ * In the meantime, this is a re-implementation for spark's use.
+ */
+object SparkRackResolver extends Logging {
+ private var dnsToSwitchMapping: DNSToSwitchMapping = _
+ private var initCalled = false
+
+ def coreResolve(conf: Configuration, hostNames: List[String]): List[Node] = {
+if (!initCalled) {
Review comment:
Meanwhile, since `var instance: SparkResolver` is static, how to maintain
more than one SparkResolvers for different `conf`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #23951: [SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
LantaoJin commented on a change in pull request #23951:
[SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
URL: https://github.com/apache/spark/pull/23951#discussion_r267179410
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala
##
@@ -17,24 +17,90 @@
package org.apache.spark.deploy.yarn
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+import com.google.common.base.Strings
import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic
+import org.apache.hadoop.net._
+import org.apache.hadoop.util.ReflectionUtils
import org.apache.hadoop.yarn.util.RackResolver
import org.apache.log4j.{Level, Logger}
+import org.apache.spark.internal.Logging
+
/**
- * Wrapper around YARN's [[RackResolver]]. This allows Spark tests to easily
override the
+ * Re-implement YARN's [[RackResolver]]. This allows Spark tests to easily
override the
* default behavior, since YARN's class self-initializes the first time it's
called, and
* future calls all use the initial configuration.
*/
-private[yarn] class SparkRackResolver {
+private[spark] class SparkRackResolver {
// RackResolver logs an INFO message whenever it resolves a rack, which is
way too often.
if (Logger.getLogger(classOf[RackResolver]).getLevel == null) {
Logger.getLogger(classOf[RackResolver]).setLevel(Level.WARN)
}
def resolve(conf: Configuration, hostName: String): String = {
-RackResolver.resolve(conf, hostName).getNetworkLocation()
+SparkRackResolver.coreResolve(conf, List(hostName)).head.getNetworkLocation
+ }
+
+ /**
+ * Added in SPARK-27038.
+ * This should be changed to `RackResolver.resolve(conf, hostNames)`
+ * in hadoop releases with YARN-9332.
+ */
+ def resolve(conf: Configuration, hostNames: List[String]): List[Node] = {
+SparkRackResolver.coreResolve(conf, hostNames)
}
+}
+/**
+ * Utility to resolve the rack for hosts in an efficient manner.
+ * It will cache the rack for individual hosts to avoid
+ * repeatedly performing the same expensive lookup.
+ *
+ * Its logic refers [[org.apache.hadoop.yarn.util.RackResolver]] and enhanced.
+ * This will be unnecessary in hadoop releases with YARN-9332.
+ * With that, we could just directly use
[[org.apache.hadoop.yarn.util.RackResolver]].
+ * In the meantime, this is a re-implementation for spark's use.
+ */
+object SparkRackResolver extends Logging {
+ private var dnsToSwitchMapping: DNSToSwitchMapping = _
+ private var initCalled = false
+
+ def coreResolve(conf: Configuration, hostNames: List[String]): List[Node] = {
+if (!initCalled) {
Review comment:
Every calling of `get` in different threads will generate a new
SparkResolver since the `conf` instances are different. So how to keep only one
cache in memory?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer
SparkQA removed a comment on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer URL: https://github.com/apache/spark/pull/24055#issuecomment-474631476 **[Test build #103698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103698/testReport)** for PR 24055 at commit [`82979d8`](https://github.com/apache/spark/commit/82979d8beb3f9a9bebc49d47123e9d35f8d25a2d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer
SparkQA commented on issue #24055: [SPARK-26820][SQL] Prints log messages when hints ignored in the analyzer URL: https://github.com/apache/spark/pull/24055#issuecomment-474676284 **[Test build #103698 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103698/testReport)** for PR 24055 at commit [`82979d8`](https://github.com/apache/spark/commit/82979d8beb3f9a9bebc49d47123e9d35f8d25a2d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default
HyukjinKwon commented on issue #24143: [SPARK-27200][WebUI][HistoryServer] History Environment tab must sort Configurations/Properties by default URL: https://github.com/apache/spark/pull/24143#issuecomment-474675080 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #24095: [SPARK-27163][PYTHON] Cleanup and consolidate Pandas UDF functionality
HyukjinKwon commented on a change in pull request #24095: [SPARK-27163][PYTHON]
Cleanup and consolidate Pandas UDF functionality
URL: https://github.com/apache/spark/pull/24095#discussion_r267178326
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/FlatMapGroupsInPandasExec.scala
##
@@ -145,7 +146,15 @@ case class FlatMapGroupsInPandasExec(
sessionLocalTimeZone,
pythonRunnerConf).compute(grouped, context.partitionId(), context)
-
columnarBatchIter.flatMap(_.rowIterator.asScala).map(UnsafeProjection.create(output,
output))
+ columnarBatchIter.flatMap { batch =>
+// Grouped Map UDF returns a StructType column in ColumnarBatch,
select the children here
+// TODO: ColumnVector getChild is protected, so use ArrowColumnVector
which is public
+val structVector = batch.column(0).asInstanceOf[ArrowColumnVector]
+val outputVectors =
output.indices.map(structVector.getChild(_).asInstanceOf[ColumnVector])
+val flattenedBatch = new ColumnarBatch(outputVectors.toArray)
+flattenedBatch.setNumRows(batch.numRows())
+flattenedBatch.rowIterator.asScala
+ }.map(UnsafeProjection.create(output, output))
Review comment:
Oh, also, let's make a separate variable for
`UnsafeProjection.create(output, output)`. I talked we talked before in my PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats
dongjoon-hyun commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats URL: https://github.com/apache/spark/pull/23803#issuecomment-474672582 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats
dongjoon-hyun commented on issue #23803: [SPARK-26894][SQL] Handle Alias as well in AggregateEstimation to propagate child stats URL: https://github.com/apache/spark/pull/23803#issuecomment-474672479 Thank you for cc-ing me, @maropu . I'll take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #24092: [SPARK-27160][SQL] Fix DecimalType when building orc filters
dongjoon-hyun closed pull request #24092: [SPARK-27160][SQL] Fix DecimalType when building orc filters URL: https://github.com/apache/spark/pull/24092 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24092: [SPARK-27160][SQL] Fix DecimalType when building orc filters
dongjoon-hyun commented on issue #24092: [SPARK-27160][SQL] Fix DecimalType when building orc filters URL: https://github.com/apache/spark/pull/24092#issuecomment-474670778 Hi, @sadhen . There is a conflict at `branch-2.4`. Could you make a PR against `branch-2.4` please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24092: [SPARK-27160][SQL] Fix DecimalType when building orc filters
dongjoon-hyun commented on issue #24092: [SPARK-27160][SQL] Fix DecimalType when building orc filters URL: https://github.com/apache/spark/pull/24092#issuecomment-474670433 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #23964: [SPARK-26975][SQL] Support nested-column pruning over limit/sample/repartition
dongjoon-hyun closed pull request #23964: [SPARK-26975][SQL] Support nested-column pruning over limit/sample/repartition URL: https://github.com/apache/spark/pull/23964 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] felixcheung commented on issue #24019: [SPARK-27099][SQL] Add 'xxhash64' for hashing arbitrary columns to Long
felixcheung commented on issue #24019: [SPARK-27099][SQL] Add 'xxhash64' for hashing arbitrary columns to Long URL: https://github.com/apache/spark/pull/24019#issuecomment-474670085 R test passes, so that part is good. someone else should review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #23964: [SPARK-26975][SQL] Support nested-column pruning over limit/sample/repartition
dongjoon-hyun commented on issue #23964: [SPARK-26975][SQL] Support nested-column pruning over limit/sample/repartition URL: https://github.com/apache/spark/pull/23964#issuecomment-474669667 After https://github.com/apache/spark/pull/23964#issuecomment-473742988 , all comments are addressed. > The current one looks good to me except for my minor comments. Since there is no further comments and this has been looking good for two days, I'll merged this. Thank you all for helping this PR, @maropu , @viirya , @dbtsai , @dilipbiswal , @cloud-fan , @gatorsmile . Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24139: [SPARK-27197][SQL][TEST] Add ReadNestedSchemaTest for file-based data sources
dongjoon-hyun commented on issue #24139: [SPARK-27197][SQL][TEST] Add ReadNestedSchemaTest for file-based data sources URL: https://github.com/apache/spark/pull/24139#issuecomment-474668400 It passes finally. Thank you all, @dbtsai , @viirya , @maropu , @HyukjinKwon . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName
AmplabJenkins removed a comment on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName URL: https://github.com/apache/spark/pull/24148#issuecomment-474664073 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName
AmplabJenkins removed a comment on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName URL: https://github.com/apache/spark/pull/24148#issuecomment-474664077 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103692/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName
AmplabJenkins commented on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName URL: https://github.com/apache/spark/pull/24148#issuecomment-474664077 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103692/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName
AmplabJenkins commented on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName URL: https://github.com/apache/spark/pull/24148#issuecomment-474664073 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName
SparkQA removed a comment on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName URL: https://github.com/apache/spark/pull/24148#issuecomment-474598749 **[Test build #103692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103692/testReport)** for PR 24148 at commit [`e9ee52f`](https://github.com/apache/spark/commit/e9ee52f5c2a3248490f473821aa10ecdfc657ccc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName
SparkQA commented on issue #24148: [MINOR][CORE] Leverage modified Utils.classForName to reduce scalastyle off for Class.forName URL: https://github.com/apache/spark/pull/24148#issuecomment-474663411 **[Test build #103692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103692/testReport)** for PR 24148 at commit [`e9ee52f`](https://github.com/apache/spark/commit/e9ee52f5c2a3248490f473821aa10ecdfc657ccc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #23912: [SPARK-21029][SS] StreamingQuery should be stopped when the SparkSession is stopped
SparkQA commented on issue #23912: [SPARK-21029][SS] StreamingQuery should be stopped when the SparkSession is stopped URL: https://github.com/apache/spark/pull/23912#issuecomment-474662998 **[Test build #103702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/103702/testReport)** for PR 23912 at commit [`d3e8fb8`](https://github.com/apache/spark/commit/d3e8fb8d05762b1a9aafbcad4c0d8998d1c400a5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24092: [SPARK-27160][SQL] Fix DecimalType when building orc filters
AmplabJenkins removed a comment on issue #24092: [SPARK-27160][SQL] Fix DecimalType when building orc filters URL: https://github.com/apache/spark/pull/24092#issuecomment-474662104 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103696/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24092: [SPARK-27160][SQL] Fix DecimalType when building orc filters
AmplabJenkins removed a comment on issue #24092: [SPARK-27160][SQL] Fix DecimalType when building orc filters URL: https://github.com/apache/spark/pull/24092#issuecomment-474662099 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
