[GitHub] [spark] AmplabJenkins removed a comment on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
AmplabJenkins removed a comment on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591290059 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
SparkQA removed a comment on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591284392 **[Test build #118952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118952/testReport)** for PR 27215 at commit [`3055d02`](https://github.com/apache/spark/commit/3055d028cce7f3a3e60e911c5d23f976c1e2df6d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
SparkQA commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591290026 **[Test build #118952 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118952/testReport)** for PR 27215 at commit [`3055d02`](https://github.com/apache/spark/commit/3055d028cce7f3a3e60e911c5d23f976c1e2df6d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
AmplabJenkins commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591290069 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118952/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
AmplabJenkins commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591290059 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
maropu commented on a change in pull request #27702: [SPARK-30955][SQL] Exclude
Generate output when aliasing in nested column pruning
URL: https://github.com/apache/spark/pull/27702#discussion_r384322324
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3396,13 +3397,24 @@ class SQLQuerySuite extends QueryTest with
SharedSparkSession with AdaptiveSpark
test("SPARK-30870: Column pruning shouldn't alias a nested column if it
means the whole " +
"structure") {
Review comment:
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yma11 commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
yma11 commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines URL: https://github.com/apache/spark/pull/27546#issuecomment-591289225 Hi @srowen and @mengxr, I have updated this PR using 256 as the value for nativaBLAS and java implementation switch. Pls take a further view. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
viirya commented on a change in pull request #27702: [SPARK-30955][SQL] Exclude
Generate output when aliasing in nested column pruning
URL: https://github.com/apache/spark/pull/27702#discussion_r384321962
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3396,13 +3397,24 @@ class SQLQuerySuite extends QueryTest with
SharedSparkSession with AdaptiveSpark
test("SPARK-30870: Column pruning shouldn't alias a nested column if it
means the whole " +
"structure") {
Review comment:
ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27567: [SPARK-30822][SQL] Remove semicolon at the end of a sql query
maropu commented on a change in pull request #27567: [SPARK-30822][SQL] Remove
semicolon at the end of a sql query
URL: https://github.com/apache/spark/pull/27567#discussion_r384321766
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
##
@@ -45,7 +45,10 @@ class SparkSqlParser(conf: SQLConf) extends
AbstractSqlParser(conf) {
private val substitutor = new VariableSubstitution(conf)
protected override def parse[T](command: String)(toResult: SqlBaseParser =>
T): T = {
-super.parse(substitutor.substitute(command))(toResult)
+val trimmed_command = if (command.trim.endsWith(";")) {
Review comment:
Can we fix this in the `SqlBase.g4` side? Then, add tests in
`PlanParserSuite`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
AmplabJenkins removed a comment on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591286742 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118951/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
SparkQA removed a comment on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591281768 **[Test build #118951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118951/testReport)** for PR 27215 at commit [`0bfa268`](https://github.com/apache/spark/commit/0bfa268f98f10e6e1e93f544c7360e70a35d4b88). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
AmplabJenkins removed a comment on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591286736 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate
maropu commented on a change in pull request #27699:
[SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate
URL: https://github.com/apache/spark/pull/27699#discussion_r384319947
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/PredicateSuite.scala
##
@@ -523,34 +523,6 @@ class PredicateSuite extends SparkFunSuite with
ExpressionEvalHelper {
}
val row0 = create_row(null)
Review comment:
Can you move this variable into the test `test("isunknown and
isnotunknown")`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
AmplabJenkins commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591286736 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
AmplabJenkins commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591286742 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118951/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
SparkQA commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591286699 **[Test build #118951 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118951/testReport)** for PR 27215 at commit [`0bfa268`](https://github.com/apache/spark/commit/0bfa268f98f10e6e1e93f544c7360e70a35d4b88). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] peter-toth commented on issue #27655: [SPARK-30897][SQL] The nullability of ArrayExists should not depend on SQLConf.get
peter-toth commented on issue #27655: [SPARK-30897][SQL] The nullability of ArrayExists should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27655#issuecomment-591285563 > @peter-toth Could you update the PR title and description? Thanks @gatorsmile, I've changed them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL
AmplabJenkins removed a comment on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL URL: https://github.com/apache/spark/pull/27691#issuecomment-591284579 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
AmplabJenkins removed a comment on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591283857 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118948/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL
AmplabJenkins removed a comment on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL URL: https://github.com/apache/spark/pull/27691#issuecomment-591284585 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118943/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL
AmplabJenkins commented on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL URL: https://github.com/apache/spark/pull/27691#issuecomment-591284585 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118943/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL
AmplabJenkins commented on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL URL: https://github.com/apache/spark/pull/27691#issuecomment-591284579 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
SparkQA commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591284392 **[Test build #118952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118952/testReport)** for PR 27215 at commit [`3055d02`](https://github.com/apache/spark/commit/3055d028cce7f3a3e60e911c5d23f976c1e2df6d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
maropu commented on a change in pull request #27702: [SPARK-30955][SQL] Exclude
Generate output when aliasing in nested column pruning
URL: https://github.com/apache/spark/pull/27702#discussion_r384317061
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3396,13 +3397,24 @@ class SQLQuerySuite extends QueryTest with
SharedSparkSession with AdaptiveSpark
test("SPARK-30870: Column pruning shouldn't alias a nested column if it
means the whole " +
"structure") {
Review comment:
nit: this is not related to this pr though, for simplicity, can we pack the
test title in a single line like `test("SPARK-30870: Column pruning shouldn't
alias a nested column for the whole structure)`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type
AmplabJenkins removed a comment on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type URL: https://github.com/apache/spark/pull/27499#issuecomment-591283341 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118944/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type
AmplabJenkins removed a comment on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type URL: https://github.com/apache/spark/pull/27499#issuecomment-591283329 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL
SparkQA removed a comment on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL URL: https://github.com/apache/spark/pull/27691#issuecomment-591215521 **[Test build #118943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118943/testReport)** for PR 27691 at commit [`c67da83`](https://github.com/apache/spark/commit/c67da839bab020e9a4da845412ec989ef862655b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type
SparkQA removed a comment on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type URL: https://github.com/apache/spark/pull/27499#issuecomment-591219315 **[Test build #118944 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118944/testReport)** for PR 27499 at commit [`68d17f7`](https://github.com/apache/spark/commit/68d17f7e71a9ca861a8939453887e71d8498f2f4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
SparkQA removed a comment on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591254985 **[Test build #118948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118948/testReport)** for PR 27702 at commit [`fec700f`](https://github.com/apache/spark/commit/fec700f67797af0514ad0ecf85b96e74829d6718). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
AmplabJenkins removed a comment on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591283850 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
SparkQA commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591283729 **[Test build #118948 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118948/testReport)** for PR 27702 at commit [`fec700f`](https://github.com/apache/spark/commit/fec700f67797af0514ad0ecf85b96e74829d6718). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL
SparkQA commented on issue #27691: [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL URL: https://github.com/apache/spark/pull/27691#issuecomment-591283862 **[Test build #118943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118943/testReport)** for PR 27691 at commit [`c67da83`](https://github.com/apache/spark/commit/c67da839bab020e9a4da845412ec989ef862655b). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class MultiplyInterval(` * `case class DivideInterval(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
AmplabJenkins commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591283850 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
AmplabJenkins commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591283857 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118948/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type
AmplabJenkins commented on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type URL: https://github.com/apache/spark/pull/27499#issuecomment-591283329 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type
AmplabJenkins commented on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type URL: https://github.com/apache/spark/pull/27499#issuecomment-591283341 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118944/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
maropu commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591283103 I found a typo in the PR description: `should should` -> a single should. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
maropu commented on a change in pull request #27702: [SPARK-30955][SQL] Exclude
Generate output when aliasing in nested column pruning
URL: https://github.com/apache/spark/pull/27702#discussion_r384315612
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -107,16 +107,17 @@ object NestedColumnAliasing {
* 1. ExtractValue -> Alias: A new alias is created for each nested field.
* 2. ExprId -> Seq[Alias]: A reference attribute has multiple aliases
pointing it.
*/
- def getAliasSubMap(exprList: Seq[Expression])
+ def getAliasSubMap(exprList: Seq[Expression], exclusiveAttrs: Seq[Attribute]
= Seq.empty)
Review comment:
Ur, I see. This is a nit comment though, I feel sharing `getAliasSubMap` in
both cases (`project w/generate case` and `simple project case`) makes the
function arguments a bit complicated. So, its better to leave some comments
about the argument `exclusiveAttrs` in the line 105.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type
SparkQA commented on issue #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type URL: https://github.com/apache/spark/pull/27499#issuecomment-591282698 **[Test build #118944 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118944/testReport)** for PR 27499 at commit [`68d17f7`](https://github.com/apache/spark/commit/68d17f7e71a9ca861a8939453887e71d8498f2f4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource
SparkQA commented on issue #27215: [SPARK-30514][K8S] add python environment support for JavaMainAppResource URL: https://github.com/apache/spark/pull/27215#issuecomment-591281768 **[Test build #118951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118951/testReport)** for PR 27215 at commit [`0bfa268`](https://github.com/apache/spark/commit/0bfa268f98f10e6e1e93f544c7360e70a35d4b88). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] peter-toth commented on issue #27675: [SPARK-30870][SQL] Column pruning shouldn't alias a nested column if it means the whole structure
peter-toth commented on issue #27675: [SPARK-30870][SQL] Column pruning shouldn't alias a nested column if it means the whole structure URL: https://github.com/apache/spark/pull/27675#issuecomment-591279917 > @peter-toth @dongjoon-hyun Can we backport this to 2.4? @gatorsmile only Spark 3 is affected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-591279473 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-591279479 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23699/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-591279479 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23699/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-591279473 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
SparkQA commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-591278980 **[Test build #118950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118950/testReport)** for PR 27617 at commit [`86ccef1`](https://github.com/apache/spark/commit/86ccef1da832d25166c3bfd018f1df9b6af5b9ee). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #27637: [SPARK-30874][SQL] Support Postgres Kerberos login in JDBC connector
HeartSaVioR commented on issue #27637: [SPARK-30874][SQL] Support Postgres Kerberos login in JDBC connector URL: https://github.com/apache/spark/pull/27637#issuecomment-591278572 ``` ./build/mvn test -Pdocker-integration-tests -pl :spark-docker-integration-tests_2.12 -Dtest=none -DwildcardSuites="org.apache.spark.sql.jdbc.*" ``` This has been failing as following error (with super slow download): ``` [ERROR] Failed to execute goal on project spark-docker-integration-tests_2.12: Could not resolve dependencies for project org.apache.spark:spark-docker-integration-tests_2.12:jar:3.0.0-SNAPSHOT: Could not transfer artifact com.ibm.db2.jcc:db2jcc4:jar:10.5.0.5 from/to db (https://app.camunda.com/nexus/content/repositories/public/): GET request of: com/ibm/db2/jcc/db2jcc4/10.5.0.5/db2jcc4-10.5.0.5.jar from db failed: Premature end of Content-Length delimited message body (expected: 3,411,524; received: 1,968,865) ``` It's not related to the change but I cannot run this to make sure the tests pass. Unfortunately I haven't found any alternative repo for db2jcc4 with such version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #27637: [SPARK-30874][SQL] Support Postgres Kerberos login in JDBC connector
HeartSaVioR edited a comment on issue #27637: [SPARK-30874][SQL] Support Postgres Kerberos login in JDBC connector URL: https://github.com/apache/spark/pull/27637#issuecomment-591278572 > ./build/mvn test -Pdocker-integration-tests -pl :spark-docker-integration-tests_2.12 -Dtest=none -DwildcardSuites="org.apache.spark.sql.jdbc.*" This has been failing as following error (with super slow download): ``` [ERROR] Failed to execute goal on project spark-docker-integration-tests_2.12: Could not resolve dependencies for project org.apache.spark:spark-docker-integration-tests_2.12:jar:3.0.0-SNAPSHOT: Could not transfer artifact com.ibm.db2.jcc:db2jcc4:jar:10.5.0.5 from/to db (https://app.camunda.com/nexus/content/repositories/public/): GET request of: com/ibm/db2/jcc/db2jcc4/10.5.0.5/db2jcc4-10.5.0.5.jar from db failed: Premature end of Content-Length delimited message body (expected: 3,411,524; received: 1,968,865) ``` It's not related to the change but I cannot run this to make sure the tests pass. Unfortunately I haven't found any alternative repo for db2jcc4 with such version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite
AmplabJenkins removed a comment on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite URL: https://github.com/apache/spark/pull/27700#issuecomment-591273302 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118941/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite
AmplabJenkins removed a comment on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite URL: https://github.com/apache/spark/pull/27700#issuecomment-591273297 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite
AmplabJenkins commented on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite URL: https://github.com/apache/spark/pull/27700#issuecomment-591273297 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite
AmplabJenkins commented on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite URL: https://github.com/apache/spark/pull/27700#issuecomment-591273302 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118941/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL]
Refactor DateTimeUtils
URL: https://github.com/apache/spark/pull/27617#discussion_r384305534
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -568,8 +563,8 @@ object DateTimeUtils {
* The result is rounded to 8 decimal places if `roundOff` is set to true.
*/
def monthsBetween(
- time1: SQLTimestamp,
- time2: SQLTimestamp,
+ time1: Long,
+ time2: Long,
Review comment:
`micros1/2`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate
beliefer commented on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate URL: https://github.com/apache/spark/pull/27699#issuecomment-591272853 cc @cloud-fan @dongjoon-hyun @maropu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite
SparkQA removed a comment on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite URL: https://github.com/apache/spark/pull/27700#issuecomment-591206045 **[Test build #118941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118941/testReport)** for PR 27700 at commit [`65b23fd`](https://github.com/apache/spark/commit/65b23fdc00b2a0650a98feea359d6bc860d1741f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite
SparkQA commented on issue #27700: [MINOR][TESTS] Use intercept instead of try-catch to assert failures in IntervalUtilsSuite URL: https://github.com/apache/spark/pull/27700#issuecomment-591272617 **[Test build #118941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118941/testReport)** for PR 27700 at commit [`65b23fd`](https://github.com/apache/spark/commit/65b23fdc00b2a0650a98feea359d6bc860d1741f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate
AmplabJenkins removed a comment on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate URL: https://github.com/apache/spark/pull/27699#issuecomment-591272311 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118942/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate
AmplabJenkins removed a comment on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate URL: https://github.com/apache/spark/pull/27699#issuecomment-591272299 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate
AmplabJenkins commented on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate URL: https://github.com/apache/spark/pull/27699#issuecomment-591272311 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118942/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate
AmplabJenkins commented on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate URL: https://github.com/apache/spark/pull/27699#issuecomment-591272299 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate
SparkQA removed a comment on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate URL: https://github.com/apache/spark/pull/27699#issuecomment-591206072 **[Test build #118942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118942/testReport)** for PR 27699 at commit [`f146322`](https://github.com/apache/spark/commit/f14632275c5a2b21032a62138c9ab5d07e4efc2b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate
SparkQA commented on issue #27699: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate URL: https://github.com/apache/spark/pull/27699#issuecomment-591271674 **[Test build #118942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118942/testReport)** for PR 27699 at commit [`f146322`](https://github.com/apache/spark/commit/f14632275c5a2b21032a62138c9ab5d07e4efc2b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #22921: [SPARK-25908][CORE][SQL] Remove old deprecated items in Spark 3
HyukjinKwon commented on a change in pull request #22921:
[SPARK-25908][CORE][SQL] Remove old deprecated items in Spark 3
URL: https://github.com/apache/spark/pull/22921#discussion_r384303852
##
File path: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
##
@@ -281,9 +265,6 @@ class GeneralMLWriter(stage: PipelineStage) extends
MLWriter with Logging {
// override for Java compatibility
override def session(sparkSession: SparkSession): this.type =
super.session(sparkSession)
-
- // override for Java compatibility
- override def context(sqlContext: SQLContext): this.type =
super.session(sqlContext.sparkSession)
Review comment:
This seems properly deprecated in `MLWriter` as its parent.
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.util.GeneralMLWriter@context(sqlContext:org.apache.spark.sql.SQLContext):GeneralMLWriter.this.type
and the Scaladoc explicitly shows this at `GeneralMLWriter` too. Seems right
to remove together if we should in `MLWriter`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #22921: [SPARK-25908][CORE][SQL] Remove old deprecated items in Spark 3
HyukjinKwon commented on a change in pull request #22921:
[SPARK-25908][CORE][SQL] Remove old deprecated items in Spark 3
URL: https://github.com/apache/spark/pull/22921#discussion_r384303852
##
File path: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
##
@@ -281,9 +265,6 @@ class GeneralMLWriter(stage: PipelineStage) extends
MLWriter with Logging {
// override for Java compatibility
override def session(sparkSession: SparkSession): this.type =
super.session(sparkSession)
-
- // override for Java compatibility
- override def context(sqlContext: SQLContext): this.type =
super.session(sqlContext.sparkSession)
Review comment:
This seems properly deprecated in `MLWriter` as its parent.
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.util.GeneralMLWriter@context(sqlContext:org.apache.spark.sql.SQLContext):GeneralMLWriter.this.type
and the Scaladoc explicitly shows this too. Seems right to remove together
if we should in `MLWriter`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #22921: [SPARK-25908][CORE][SQL] Remove old deprecated items in Spark 3
HyukjinKwon commented on a change in pull request #22921:
[SPARK-25908][CORE][SQL] Remove old deprecated items in Spark 3
URL: https://github.com/apache/spark/pull/22921#discussion_r384303852
##
File path: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
##
@@ -281,9 +265,6 @@ class GeneralMLWriter(stage: PipelineStage) extends
MLWriter with Logging {
// override for Java compatibility
override def session(sparkSession: SparkSession): this.type =
super.session(sparkSession)
-
- // override for Java compatibility
- override def context(sqlContext: SQLContext): this.type =
super.session(sqlContext.sparkSession)
Review comment:
This seems properly deprecated in `MLWriter` as its parent.
and the Scaladoc explicitly shows this too. Seems right to remove together
if we should in `MLWriter`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL]
Refactor DateTimeUtils
URL: https://github.com/apache/spark/pull/27617#discussion_r384303953
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -39,84 +39,75 @@ import org.apache.spark.unsafe.types.{CalendarInterval,
UTF8String}
*/
object DateTimeUtils {
- // we use Int and Long internally to represent [[DateType]] and
[[TimestampType]]
- type SQLDate = Int
- type SQLTimestamp = Long
-
// see
http://stackoverflow.com/questions/466321/convert-unix-timestamp-to-julian
// it's 2440587.5, rounding up to compatible with Hive
final val JULIAN_DAY_OF_EPOCH = 2440588
- final val TimeZoneGMT = TimeZone.getTimeZone("GMT")
- final val TimeZoneUTC = TimeZone.getTimeZone("UTC")
-
val TIMEZONE_OPTION = "timeZone"
- def defaultTimeZone(): TimeZone = TimeZone.getDefault()
-
def getZoneId(timeZoneId: String): ZoneId = ZoneId.of(timeZoneId,
ZoneId.SHORT_IDS)
def getTimeZone(timeZoneId: String): TimeZone = {
TimeZone.getTimeZone(getZoneId(timeZoneId))
}
- def microsToDays(timestamp: SQLTimestamp): SQLDate = {
-microsToDays(timestamp, defaultTimeZone().toZoneId)
+ def microsToDays(timestamp: Long): Int = {
+microsToDays(timestamp, ZoneId.systemDefault())
}
- def microsToDays(timestamp: SQLTimestamp, zoneId: ZoneId): SQLDate = {
+ def microsToDays(timestamp: Long, zoneId: ZoneId): Int = {
val instant = microsToInstant(timestamp)
localDateToDays(LocalDateTime.ofInstant(instant, zoneId).toLocalDate)
}
- def daysToMicros(days: SQLDate): SQLTimestamp = {
-daysToMicros(days, defaultTimeZone().toZoneId)
+ def daysToMicros(days: Int): Long = {
+daysToMicros(days, ZoneId.systemDefault())
}
- def daysToMicros(days: SQLDate, zoneId: ZoneId): SQLTimestamp = {
+ def daysToMicros(days: Int, zoneId: ZoneId): Long = {
val instant = daysToLocalDate(days).atStartOfDay(zoneId).toInstant
instantToMicros(instant)
}
- // Converts Timestamp to string according to Hive TimestampWritable
convention.
- def timestampToString(tf: TimestampFormatter, us: SQLTimestamp): String = {
-tf.format(us)
+ // Converts the `micros` timestamp to string according to Hive
TimestampWritable convention.
+ def timestampToString(tf: TimestampFormatter, micros: Long): String = {
+tf.format(micros)
}
/**
* Returns the number of days since epoch from java.sql.Date.
*/
- def fromJavaDate(date: Date): SQLDate = {
+ def fromJavaDate(date: Date): Int = {
microsToDays(millisToMicros(date.getTime))
}
/**
* Returns a java.sql.Date from number of days since epoch.
*/
- def toJavaDate(daysSinceEpoch: SQLDate): Date = {
+ def toJavaDate(daysSinceEpoch: Int): Date = {
Review comment:
`days` for consistency?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL]
Refactor DateTimeUtils
URL: https://github.com/apache/spark/pull/27617#discussion_r384303769
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -39,84 +39,75 @@ import org.apache.spark.unsafe.types.{CalendarInterval,
UTF8String}
*/
object DateTimeUtils {
- // we use Int and Long internally to represent [[DateType]] and
[[TimestampType]]
- type SQLDate = Int
- type SQLTimestamp = Long
-
// see
http://stackoverflow.com/questions/466321/convert-unix-timestamp-to-julian
// it's 2440587.5, rounding up to compatible with Hive
final val JULIAN_DAY_OF_EPOCH = 2440588
- final val TimeZoneGMT = TimeZone.getTimeZone("GMT")
- final val TimeZoneUTC = TimeZone.getTimeZone("UTC")
-
val TIMEZONE_OPTION = "timeZone"
- def defaultTimeZone(): TimeZone = TimeZone.getDefault()
-
def getZoneId(timeZoneId: String): ZoneId = ZoneId.of(timeZoneId,
ZoneId.SHORT_IDS)
def getTimeZone(timeZoneId: String): TimeZone = {
TimeZone.getTimeZone(getZoneId(timeZoneId))
}
- def microsToDays(timestamp: SQLTimestamp): SQLDate = {
-microsToDays(timestamp, defaultTimeZone().toZoneId)
+ def microsToDays(timestamp: Long): Int = {
+microsToDays(timestamp, ZoneId.systemDefault())
}
- def microsToDays(timestamp: SQLTimestamp, zoneId: ZoneId): SQLDate = {
+ def microsToDays(timestamp: Long, zoneId: ZoneId): Int = {
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL]
Refactor DateTimeUtils
URL: https://github.com/apache/spark/pull/27617#discussion_r384303728
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -39,84 +39,75 @@ import org.apache.spark.unsafe.types.{CalendarInterval,
UTF8String}
*/
object DateTimeUtils {
- // we use Int and Long internally to represent [[DateType]] and
[[TimestampType]]
- type SQLDate = Int
- type SQLTimestamp = Long
-
// see
http://stackoverflow.com/questions/466321/convert-unix-timestamp-to-julian
// it's 2440587.5, rounding up to compatible with Hive
final val JULIAN_DAY_OF_EPOCH = 2440588
- final val TimeZoneGMT = TimeZone.getTimeZone("GMT")
- final val TimeZoneUTC = TimeZone.getTimeZone("UTC")
-
val TIMEZONE_OPTION = "timeZone"
- def defaultTimeZone(): TimeZone = TimeZone.getDefault()
-
def getZoneId(timeZoneId: String): ZoneId = ZoneId.of(timeZoneId,
ZoneId.SHORT_IDS)
def getTimeZone(timeZoneId: String): TimeZone = {
TimeZone.getTimeZone(getZoneId(timeZoneId))
}
- def microsToDays(timestamp: SQLTimestamp): SQLDate = {
-microsToDays(timestamp, defaultTimeZone().toZoneId)
+ def microsToDays(timestamp: Long): Int = {
Review comment:
shall we make the parameter name consistent? `micros`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite
AmplabJenkins removed a comment on issue #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite URL: https://github.com/apache/spark/pull/27703#issuecomment-591268157 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite
SparkQA commented on issue #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite URL: https://github.com/apache/spark/pull/27703#issuecomment-591269914 **[Test build #118949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118949/testReport)** for PR 27703 at commit [`412062d`](https://github.com/apache/spark/commit/412062d37274251c62ab82a379f1208bfdf28a53). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite
AmplabJenkins removed a comment on issue #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite URL: https://github.com/apache/spark/pull/27703#issuecomment-591268165 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23698/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite
AmplabJenkins commented on issue #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite URL: https://github.com/apache/spark/pull/27703#issuecomment-591268157 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite
AmplabJenkins commented on issue #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite URL: https://github.com/apache/spark/pull/27703#issuecomment-591268165 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23698/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite
HyukjinKwon opened a new pull request #27703: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite URL: https://github.com/apache/spark/pull/27703 ### What changes were proposed in this pull request? After https://github.com/apache/spark/pull/27387, the tests below fail specifically in one job https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/ in Jenkins ``` org.apache.spark.sql.hive.HiveShowCreateTableSuite.simple hive table org.apache.spark.sql.hive.HiveShowCreateTableSuite.simple external hive table org.apache.spark.sql.hive.HiveShowCreateTableSuite.hive bucketing is supported ``` The profile is same as PR builder but seems it fails specifically in this machine. See https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/278/ as an example. Seems the legacy configuration `spark.sql.legacy.createHiveTableByDefault.enabled` is not being set due to the inconsistency between `SQLConf.get` and the active Spark session described in the https://github.com/apache/spark/pull/27387. This PR proposes to explicitly set the configuration against the session used instead of `SQLConf.get`. ### Why are the changes needed? To make `spark-master-test-sbt-hadoop-2.7-hive-2.3` job pass. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Cannot reproduce in my local. Presumably it cannot be reproduced in the PR builder. We should see if the tests pass after this PR is merged This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #27632: [SPARK-30872][SQL] Constraints inferred from inferred attributes
wangyum commented on a change in pull request #27632: [SPARK-30872][SQL]
Constraints inferred from inferred attributes
URL: https://github.com/apache/spark/pull/27632#discussion_r38128
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
##
@@ -322,4 +322,13 @@ class DataFrameJoinSuite extends QueryTest
}
}
}
+
+ test("SPARK-30872: Constraints inferred from inferred attributes") {
+withTable("t1") {
+ spark.range(20).selectExpr("id as a", "id as b", "id as
c").write.saveAsTable("t1")
+ checkAnswer(
+spark.sql("select count(*) from t1 where a = b and b = c and (c = 3 or
c = 13)"),
+Row(2) :: Nil)
+}
+ }
Review comment:
It will throw `TreeNodeException: Once strategy's idempotence is broken for
batch Infer Filters` before this PR:
```java
[info] - SPARK-30872: Constraints inferred from inferred attributes ***
FAILED *** (222 milliseconds)
[info] org.apache.spark.sql.catalyst.errors.package$TreeNodeException:
Once strategy's idempotence is broken for batch Infer Filters
[info] Aggregate [count(1) AS count(1)#5842L]
Aggregate [count(1) AS count(1)#5842L]
[info] +- Project
+- Project
[info] ! +- Filter ((a#5839L = c#5841L) AND isnotnull(b#5840L)) AND
isnotnull(c#5841L)) AND ((b#5840L = 3) OR (b#5840L = 13))) AND
isnotnull(a#5839L)) AND (((a#5839L = b#5840L) AND (b#5840L = c#5841L)) AND
((c#5841L = 3) OR (c#5841L = 13 +- Filter (((a#5839L = 3) OR (a#5839L
= 13)) AND ((a#5839L = c#5841L) AND isnotnull(b#5840L)) AND
isnotnull(c#5841L)) AND ((b#5840L = 3) OR (b#5840L = 13))) AND
isnotnull(a#5839L)) AND (((a#5839L = b#5840L) AND (b#5840L = c#5841L)) AND
((c#5841L = 3) OR (c#5841L = 13)
[info]+- Relation[a#5839L,b#5840L,c#5841L] parquet
+- Relation[a#5839L,b#5840L,c#5841L] parquet
[info] , tree:
[info] Aggregate [count(1) AS count(1)#5842L]
[info] +- Project
[info]+- Filter (((a#5839L = 3) OR (a#5839L = 13)) AND ((a#5839L =
c#5841L) AND isnotnull(b#5840L)) AND isnotnull(c#5841L)) AND ((b#5840L = 3) OR
(b#5840L = 13))) AND isnotnull(a#5839L)) AND (((a#5839L = b#5840L) AND (b#5840L
= c#5841L)) AND ((c#5841L = 3) OR (c#5841L = 13)
[info] +- Relation[a#5839L,b#5840L,c#5841L] parquet
[info] at
org.apache.spark.sql.catalyst.rules.RuleExecutor.checkBatchIdempotence(RuleExecutor.scala:100)
[info] at
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:187)
[info] at
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:132)
[info] at scala.collection.immutable.List.foreach(List.scala:392)
[info] at
org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:132)
[info] at
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:111)
[info] at
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
[info] at
org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:111)
[info] at
org.apache.spark.sql.execution.QueryExecution.$anonfun$optimizedPlan$1(QueryExecution.scala:82)
[info] at
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
[info] at
org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:119)
[info] at
org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:762)
[info] at
org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:119)
[info] at
org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:82)
[info] at
org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:79)
[info] at
org.apache.spark.sql.QueryTest.assertEmptyMissingInput(QueryTest.scala:231)
[info] at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:154)
[info] at
org.apache.spark.sql.DataFrameJoinSuite.$anonfun$new$51(DataFrameJoinSuite.scala:332)
[info] at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[info]
[GitHub] [spark] zhengruifeng commented on a change in pull request #22921: [SPARK-25908][CORE][SQL] Remove old deprecated items in Spark 3
zhengruifeng commented on a change in pull request #22921:
[SPARK-25908][CORE][SQL] Remove old deprecated items in Spark 3
URL: https://github.com/apache/spark/pull/22921#discussion_r384292322
##
File path: mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
##
@@ -281,9 +265,6 @@ class GeneralMLWriter(stage: PipelineStage) extends
MLWriter with Logging {
// override for Java compatibility
override def session(sparkSession: SparkSession): this.type =
super.session(sparkSession)
-
- // override for Java compatibility
- override def context(sqlContext: SQLContext): this.type =
super.session(sqlContext.sparkSession)
Review comment:
@srowen This public method seems had not been deprecated before removal, and
is avaiable in 2.4.5.
```
scala> import org.apache.spark.ml.util.GeneralMLWriter
import org.apache.spark.ml.util.GeneralMLWriter
scala> new GeneralMLWriter(null).context(spark.sqlContext)
res0: org.apache.spark.ml.util.GeneralMLWriter =
org.apache.spark.ml.util.GeneralMLWriter@26b150cd
```
There is no deprecation warning above. Does it matter?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
MaxGekk commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-591257216 @cloud-fan Please, have a look at this if you have time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27676: [SPARK-30925][SQL] Prevent overflow/round errors in conversions of milliseconds to/from microseconds
MaxGekk commented on issue #27676: [SPARK-30925][SQL] Prevent overflow/round errors in conversions of milliseconds to/from microseconds URL: https://github.com/apache/spark/pull/27676#issuecomment-591256300 > Is it a behavior change? @gatorsmile Yes, this can change results but previous behavior is wrong, most likely. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
AmplabJenkins commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591255348 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
AmplabJenkins removed a comment on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591255361 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23697/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
AmplabJenkins removed a comment on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591255348 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
AmplabJenkins commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591255361 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23697/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
SparkQA commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591254985 **[Test build #118948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118948/testReport)** for PR 27702 at commit [`fec700f`](https://github.com/apache/spark/commit/fec700f67797af0514ad0ecf85b96e74829d6718). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
viirya commented on issue #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702#issuecomment-591253434 cc @dongjoon-hyun @peter-toth @maropu @HyukjinKwon @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya opened a new pull request #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning
viirya opened a new pull request #27702: [SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning URL: https://github.com/apache/spark/pull/27702 ### What changes were proposed in this pull request? When aliasing in nested column pruning in Project on top of Generate, we should exclude Generate outputs. ### Why are the changes needed? Right now we would prune nested columns in Project on top of Generate. It is possible that referred nested columns are from Generate's outputs, not from its child. To address that case, we should should exclude Generate outputs when aliasing in nested column pruning. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Unit test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables.
AmplabJenkins removed a comment on issue #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables. URL: https://github.com/apache/spark/pull/27532#issuecomment-591251202 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23696/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables.
AmplabJenkins removed a comment on issue #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables. URL: https://github.com/apache/spark/pull/27532#issuecomment-591251190 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables.
AmplabJenkins commented on issue #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables. URL: https://github.com/apache/spark/pull/27532#issuecomment-591251190 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables.
AmplabJenkins commented on issue #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables. URL: https://github.com/apache/spark/pull/27532#issuecomment-591251202 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23696/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables.
imback82 commented on a change in pull request #27532: [SPARK-30782][SQL]
Column resolution doesn't respect current catalog/namespace for v2 tables.
URL: https://github.com/apache/spark/pull/27532#discussion_r384284740
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
##
@@ -700,19 +709,29 @@ class DataSourceV2SQLSuite
}
test("qualified column names for v1 tables") {
-// unset this config to use the default v2 session catalog.
-spark.conf.unset(V2_SESSION_CATALOG_IMPLEMENTATION.key)
-
-withTable("t") {
- sql("CREATE TABLE t USING json AS SELECT 1 AS i")
- checkAnswer(sql("select t.i from spark_catalog.default.t"), Row(1))
- checkAnswer(sql("select default.t.i from spark_catalog.default.t"),
Row(1))
+Seq(true, false).foreach { useV1Table =>
+ val format = if (useV1Table) "json" else v2Format
+ if (useV1Table) {
+spark.conf.unset(V2_SESSION_CATALOG_IMPLEMENTATION.key)
+ } else {
+spark.conf.set(
+ V2_SESSION_CATALOG_IMPLEMENTATION.key,
classOf[InMemoryTableSessionCatalog].getName)
+ }
- // catalog name cannot be used for v1 tables.
- val ex = intercept[AnalysisException] {
-sql(s"select spark_catalog.default.t.i from spark_catalog.default.t")
+ withTable("t") {
+sql(s"CREATE TABLE t USING $format AS SELECT 1 AS i")
+checkAnswer(sql("select i from t"), Row(1))
+checkAnswer(sql("select t.i from t"), Row(1))
+checkAnswer(sql("select default.t.i from t"), Row(1))
+checkAnswer(sql("select t.i from spark_catalog.default.t"), Row(1))
+checkAnswer(sql("select default.t.i from spark_catalog.default.t"),
Row(1))
+
+// catalog name cannot be used for v1 tables.
Review comment:
Thanks, updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables.
SparkQA commented on issue #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables. URL: https://github.com/apache/spark/pull/27532#issuecomment-591250873 **[Test build #118947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118947/testReport)** for PR 27532 at commit [`f23ded6`](https://github.com/apache/spark/commit/f23ded6c07d00091ae21f0b04641743a3e6acb86). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] winningsix commented on a change in pull request #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
winningsix commented on a change in pull request #27546:
[SPARK-30773][ML]Support NativeBlas for level-1 routines
URL: https://github.com/apache/spark/pull/27546#discussion_r384282022
##
File path: mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala
##
@@ -27,15 +27,35 @@ private[spark] object BLAS extends Serializable {
@transient private var _f2jBLAS: NetlibBLAS = _
@transient private var _nativeBLAS: NetlibBLAS = _
+ @transient private val vectorSizeThreshold: Int = 256
Review comment:
Hint: Make this final and static
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions URL: https://github.com/apache/spark/pull/27580#issuecomment-591246463 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23695/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
AmplabJenkins removed a comment on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions URL: https://github.com/apache/spark/pull/27580#issuecomment-591246456 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions URL: https://github.com/apache/spark/pull/27580#issuecomment-591246463 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23695/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
AmplabJenkins commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions URL: https://github.com/apache/spark/pull/27580#issuecomment-591246456 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
SparkQA commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions URL: https://github.com/apache/spark/pull/27580#issuecomment-591246166 **[Test build #118946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118946/testReport)** for PR 27580 at commit [`8dfee67`](https://github.com/apache/spark/commit/8dfee675f6bfaf04068b856fc15158cea307f188). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables.
cloud-fan commented on a change in pull request #27532: [SPARK-30782][SQL]
Column resolution doesn't respect current catalog/namespace for v2 tables.
URL: https://github.com/apache/spark/pull/27532#discussion_r384279065
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
##
@@ -700,19 +709,29 @@ class DataSourceV2SQLSuite
}
test("qualified column names for v1 tables") {
-// unset this config to use the default v2 session catalog.
-spark.conf.unset(V2_SESSION_CATALOG_IMPLEMENTATION.key)
-
-withTable("t") {
- sql("CREATE TABLE t USING json AS SELECT 1 AS i")
- checkAnswer(sql("select t.i from spark_catalog.default.t"), Row(1))
- checkAnswer(sql("select default.t.i from spark_catalog.default.t"),
Row(1))
+Seq(true, false).foreach { useV1Table =>
+ val format = if (useV1Table) "json" else v2Format
+ if (useV1Table) {
+spark.conf.unset(V2_SESSION_CATALOG_IMPLEMENTATION.key)
+ } else {
+spark.conf.set(
+ V2_SESSION_CATALOG_IMPLEMENTATION.key,
classOf[InMemoryTableSessionCatalog].getName)
+ }
- // catalog name cannot be used for v1 tables.
- val ex = intercept[AnalysisException] {
-sql(s"select spark_catalog.default.t.i from spark_catalog.default.t")
+ withTable("t") {
+sql(s"CREATE TABLE t USING $format AS SELECT 1 AS i")
+checkAnswer(sql("select i from t"), Row(1))
+checkAnswer(sql("select t.i from t"), Row(1))
+checkAnswer(sql("select default.t.i from t"), Row(1))
+checkAnswer(sql("select t.i from spark_catalog.default.t"), Row(1))
+checkAnswer(sql("select default.t.i from spark_catalog.default.t"),
Row(1))
+
+// catalog name cannot be used for v1 tables.
Review comment:
`v1 tables` -> `tables in the session catalog`, as we are testing both v1
and v2 tables here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables.
cloud-fan commented on a change in pull request #27532: [SPARK-30782][SQL]
Column resolution doesn't respect current catalog/namespace for v2 tables.
URL: https://github.com/apache/spark/pull/27532#discussion_r384278975
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
##
@@ -700,19 +709,29 @@ class DataSourceV2SQLSuite
}
test("qualified column names for v1 tables") {
-// unset this config to use the default v2 session catalog.
-spark.conf.unset(V2_SESSION_CATALOG_IMPLEMENTATION.key)
-
-withTable("t") {
- sql("CREATE TABLE t USING json AS SELECT 1 AS i")
- checkAnswer(sql("select t.i from spark_catalog.default.t"), Row(1))
- checkAnswer(sql("select default.t.i from spark_catalog.default.t"),
Row(1))
+Seq(true, false).foreach { useV1Table =>
+ val format = if (useV1Table) "json" else v2Format
+ if (useV1Table) {
+spark.conf.unset(V2_SESSION_CATALOG_IMPLEMENTATION.key)
Review comment:
shall we keep this comment: `// unset this config to use the default v2
session catalog.`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables.
cloud-fan commented on a change in pull request #27532: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables. URL: https://github.com/apache/spark/pull/27532#discussion_r384278366 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -933,7 +935,7 @@ class Analyzer( v1SessionCatalog.getRelation(v1Table.v1Table) case table => SubqueryAlias( -identifier, +ident.asMultipartIdentifier, Review comment: ah ok. Let's leave it for now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
