[GitHub] [spark] xccui commented on pull request #28523: [SPARK-31706][SQL] add back the support of streaming update mode
xccui commented on pull request #28523: URL: https://github.com/apache/spark/pull/28523#issuecomment-629043813 @HeartSaVioR Yes. It seems the output mode option was mainly designed for stateful aggregations, which means it actually works in a restricted way. Ideally, to support complete mode, all the operators must be capable of outputting the "complete" result seen so far for each epoch. Personally, I'm in favor of removing this mode in a future version. But for now, I propose to add more restrictions while doing the plan check (e.g., disallowing the union situation you mentioned) and also a note to the documentation. IMO, the mode of the result table should only be decided by the operators in the plan and it could either be "append" or "update" (including the current "complete" mode). Basically, the designated sink should match the mode of the result table. Usually, supporting "update" needs more effort and that means only part of the sinks could be chosen for a plan containing an aggregation or some kind of joins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
yaooqinn commented on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629040211 cc @cloud-fan @HeartSaVioR thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #25870: [SPARK-27936][K8S] Support python deps
dongjoon-hyun commented on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629039005 It's still assigned to `amp-jenkins-worker-05`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
AmplabJenkins removed a comment on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629038579 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
AmplabJenkins commented on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629038579 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
SparkQA commented on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629038184 **[Test build #122653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122653/testReport)** for PR 28537 at commit [`97597ac`](https://github.com/apache/spark/commit/97597ac2d958590eb6bacfc768ad023019b12ded). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #25870: [SPARK-27936][K8S] Support python deps
AmplabJenkins removed a comment on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629036880 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122650/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #25870: [SPARK-27936][K8S] Support python deps
SparkQA commented on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629036862 **[Test build #122650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122650/testReport)** for PR 25870 at commit [`a9f055b`](https://github.com/apache/spark/commit/a9f055b4642d487b724a628c4d6f682c71fd2d2c). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #25870: [SPARK-27936][K8S] Support python deps
AmplabJenkins removed a comment on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629036875 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #25870: [SPARK-27936][K8S] Support python deps
SparkQA removed a comment on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629033573 **[Test build #122650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122650/testReport)** for PR 25870 at commit [`a9f055b`](https://github.com/apache/spark/commit/a9f055b4642d487b724a628c4d6f682c71fd2d2c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #25870: [SPARK-27936][K8S] Support python deps
AmplabJenkins commented on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629036875 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
AmplabJenkins removed a comment on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-629036335 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28509: [SPARK-31655][BUILD][3.0] Upgrade snappy-java to 1.1.7.5
AmplabJenkins commented on pull request #28509: URL: https://github.com/apache/spark/pull/28509#issuecomment-629036386 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
AmplabJenkins commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-629036335 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28509: [SPARK-31655][BUILD][3.0] Upgrade snappy-java to 1.1.7.5
AmplabJenkins removed a comment on pull request #28509: URL: https://github.com/apache/spark/pull/28509#issuecomment-629036386 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
SparkQA commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-629035880 **[Test build #122651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122651/testReport)** for PR 28528 at commit [`52c22ce`](https://github.com/apache/spark/commit/52c22ce34cf95b85e327391e554ea1f7c588dbe7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28509: [SPARK-31655][BUILD][3.0] Upgrade snappy-java to 1.1.7.5
SparkQA commented on pull request #28509: URL: https://github.com/apache/spark/pull/28509#issuecomment-629035863 **[Test build #122652 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122652/testReport)** for PR 28509 at commit [`692db6a`](https://github.com/apache/spark/commit/692db6aee585af63ce57cd22733b78beaf3d1b51). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28509: [SPARK-31655][BUILD][3.0] Upgrade snappy-java to 1.1.7.5
dongjoon-hyun commented on pull request #28509: URL: https://github.com/apache/spark/pull/28509#issuecomment-629035653 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
dongjoon-hyun commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-629035487 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
dongjoon-hyun edited a comment on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629034425 Ya. The failure jobs are on the same broken Jenkins machine, `amp-jenkins-worker-05`. And, the running jobs are on `amp-jenkins-worker-03` and `amp-jenkins-worker-04`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
dongjoon-hyun commented on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629034425 Ya. The failure jobs are on the same broken Jenkins machine, `amp-jenkins-worker-05`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
AmplabJenkins commented on pull request #28517: URL: https://github.com/apache/spark/pull/28517#issuecomment-629033855 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
AmplabJenkins removed a comment on pull request #28517: URL: https://github.com/apache/spark/pull/28517#issuecomment-629033855 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #25870: [SPARK-27936][K8S] Support python deps
SparkQA commented on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629033573 **[Test build #122650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122650/testReport)** for PR 25870 at commit [`a9f055b`](https://github.com/apache/spark/commit/a9f055b4642d487b724a628c4d6f682c71fd2d2c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
SparkQA commented on pull request #28517: URL: https://github.com/apache/spark/pull/28517#issuecomment-629033574 **[Test build #122649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122649/testReport)** for PR 28517 at commit [`84c172f`](https://github.com/apache/spark/commit/84c172fb20321b082fc98fa419f9756ec44a6a7e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
HeartSaVioR commented on a change in pull request #28517:
URL: https://github.com/apache/spark/pull/28517#discussion_r425571232
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##
@@ -2706,33 +2706,6 @@ class HiveDDLSuite
}
}
- test("SPARK-30098: create table without provider should " +
Review comment:
These two tests are depending on the config, hence I'd rather not
restore these tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #25870: [SPARK-27936][K8S] Support python deps
dongjoon-hyun commented on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629032842 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #28501: [SPARK-31663][SQL] Grouping sets with having clause returns the wrong result
xuanyuanking commented on a change in pull request #28501:
URL: https://github.com/apache/spark/pull/28501#discussion_r425571365
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##
@@ -629,12 +629,7 @@ class AstBuilder(conf: SQLConf) extends
SqlBaseBaseVisitor[AnyRef] with Logging
case p: Predicate => p
case e => Cast(e, BooleanType)
}
-plan match {
- case aggregate: Aggregate =>
-AggregateWithHaving(predicate, aggregate)
- case _ =>
-Filter(predicate, plan)
-}
+UnresolvedHaving(predicate, plan)
Review comment:
Yes, it still works, the `UnresolvedHaving` will be changed to Filter in
rule `ResolveAggregateFunction`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #25870: [SPARK-27936][K8S] Support python deps
AmplabJenkins removed a comment on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629031989 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122647/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #25870: [SPARK-27936][K8S] Support python deps
SparkQA removed a comment on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629028801 **[Test build #122647 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122647/testReport)** for PR 25870 at commit [`a9f055b`](https://github.com/apache/spark/commit/a9f055b4642d487b724a628c4d6f682c71fd2d2c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #25870: [SPARK-27936][K8S] Support python deps
SparkQA commented on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629031965 **[Test build #122647 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122647/testReport)** for PR 25870 at commit [`a9f055b`](https://github.com/apache/spark/commit/a9f055b4642d487b724a628c4d6f682c71fd2d2c). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #25870: [SPARK-27936][K8S] Support python deps
AmplabJenkins commented on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629031981 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #25870: [SPARK-27936][K8S] Support python deps
AmplabJenkins removed a comment on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629031981 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28501: [SPARK-31663][SQL] Grouping sets with having clause returns the wrong result
AmplabJenkins commented on pull request #28501: URL: https://github.com/apache/spark/pull/28501#issuecomment-629031566 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #28501: [SPARK-31663][SQL] Grouping sets with having clause returns the wrong result
xuanyuanking commented on a change in pull request #28501:
URL: https://github.com/apache/spark/pull/28501#discussion_r425570419
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -2163,22 +2235,29 @@ class Analyzer(
alias.toAttribute
}
}
-
- // Push the aggregate expressions into the aggregate (if any).
- if (aggregateExpressions.nonEmpty) {
-Project(agg.output,
- Filter(transformedAggregateFilter,
-agg.copy(aggregateExpressions = agg.aggregateExpressions ++
aggregateExpressions)))
- } else {
-filter
- }
+ (aggregateExpressions, Some(transformedAggregateFilter))
} else {
- filter
+ (aggregateExpressions, None)
}
} catch {
// Attempting to resolve in the aggregate can result in ambiguity.
When this happens,
// just return the original plan.
-case ae: AnalysisException => filter
+case ae: AnalysisException => (aggregateExpressions, None)
+ }
+}
+
+def resolveHaving(filter: Filter, agg: Aggregate): LogicalPlan = {
+ // Try resolving the condition of the filter as though it is in the
aggregate clause
+ val (aggregateExpressions, resolvedHavingCond) =
+resolveFilterCondInAggregate(filter.condition, agg)
+
+ // Push the aggregate expressions into the aggregate (if any).
+ if (aggregateExpressions.nonEmpty) {
Review comment:
Copy, done in 3b48e38.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #28501: [SPARK-31663][SQL] Grouping sets with having clause returns the wrong result
xuanyuanking commented on a change in pull request #28501:
URL: https://github.com/apache/spark/pull/28501#discussion_r425570296
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -2125,13 +2196,14 @@ class Analyzer(
condition.find(_.isInstanceOf[AggregateExpression]).isDefined
}
-def resolveHaving(filter: Filter, agg: Aggregate): LogicalPlan = {
- // Try resolving the condition of the filter as though it is in the
aggregate clause
+def resolveFilterCondInAggregate(
+filterCond: Expression, agg: Aggregate): (Seq[NamedExpression],
Option[Expression]) = {
Review comment:
Thanks, done in 3b48e38.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28501: [SPARK-31663][SQL] Grouping sets with having clause returns the wrong result
AmplabJenkins removed a comment on pull request #28501: URL: https://github.com/apache/spark/pull/28501#issuecomment-629031566 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #28501: [SPARK-31663][SQL] Grouping sets with having clause returns the wrong result
xuanyuanking commented on a change in pull request #28501: URL: https://github.com/apache/spark/pull/28501#discussion_r425570468 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala ## @@ -540,11 +540,12 @@ case class UnresolvedOrdinal(ordinal: Int) } /** - * Represents unresolved aggregate with having clause, it is turned by the analyzer into a Filter. + * Represents unresolved having clause, the child for it can be Aggregate, Grouping Sets, Rollup Review comment: Thanks, done in 3b48e38. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #28501: [SPARK-31663][SQL] Grouping sets with having clause returns the wrong result
xuanyuanking commented on a change in pull request #28501:
URL: https://github.com/apache/spark/pull/28501#discussion_r425570127
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -595,8 +601,73 @@ class Analyzer(
}
}
-// This require transformUp to replace grouping()/grouping_id() in
resolved Filter/Sort
-def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
+private def tryResolveHavingCondition(
+a: UnresolvedHaving, havingCond: Expression, agg: LogicalPlan):
LogicalPlan = {
+ val aggForResolving = agg match {
+// For CUBE/ROLLUP expressions, to avoid resolving repeatedly, here we
delete them from
+// groupingExpressions for condition resolving.
+case a @ Aggregate(Seq(c @ Cube(groupByExprs)), _, _) =>
+ a.copy(groupingExpressions = groupByExprs)
+case a @ Aggregate(Seq(r @ Rollup(groupByExprs)), _, _) =>
+ a.copy(groupingExpressions = groupByExprs)
+case g: GroupingSets =>
+ Aggregate(
+getFinalGroupByExpressions(g.selectedGroupByExprs, g.groupByExprs),
+g.aggregations, g.child)
+ }
+ // Try resolving the condition of the filter as though it is in the
aggregate clause
+ val (extraAggExprs, resolvedHavingCond) =
+ResolveAggregateFunctions.resolveFilterCondInAggregate(havingCond,
aggForResolving)
+
+ // Push the aggregate expressions into the aggregate (if any).
+ if (extraAggExprs.nonEmpty) {
+val newChild = agg match {
+ case Aggregate(Seq(c @ Cube(groupByExprs)), aggregateExpressions,
child) =>
+constructAggregate(
+ cubeExprs(groupByExprs), groupByExprs, aggregateExpressions ++
extraAggExprs, child)
+ case Aggregate(Seq(r @ Rollup(groupByExprs)), aggregateExpressions,
child) =>
+constructAggregate(
+ rollupExprs(groupByExprs), groupByExprs, aggregateExpressions ++
extraAggExprs, child)
+ case x: GroupingSets =>
+constructAggregate(
+ x.selectedGroupByExprs, x.groupByExprs, x.aggregations ++
extraAggExprs, x.child)
+}
+
+// Since the exprId of extraAggExprs will be changed in the
constructed aggregate, and the
+// aggregateExpressions keeps the input order. So here we build an
exprMap to resolve the
+// condition again.
+val exprMap = extraAggExprs.zip(
+ newChild.asInstanceOf[Aggregate].aggregateExpressions.takeRight(
+extraAggExprs.length)).toMap
+val newCond = resolvedHavingCond.get.transform {
+ case ne: NamedExpression if exprMap.contains(ne) => exprMap(ne)
+}
+Project(newChild.output.dropRight(extraAggExprs.length),
+ Filter(newCond, newChild))
+ } else {
+a
+ }
+}
+
+// This require transformDown to resolve having condition when generating
aggregate node for
+// CUBE/ROLLUP/GROUPING SETS. This also replace grouping()/grouping_id()
in resolved
+// Filter/Sort.
+def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsDown {
+ case a @ UnresolvedHaving(
+ havingCondition, agg @ Aggregate(Seq(c @ Cube(groupByExprs)),
aggregateExpressions, _))
+ if agg.childrenResolved &&
!havingCondition.isInstanceOf[SubqueryExpression]
Review comment:
Thanks for reminding. Yes, we don't need this. Deleted in 3b48e38.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #28501: [SPARK-31663][SQL] Grouping sets with having clause returns the wrong result
xuanyuanking commented on a change in pull request #28501:
URL: https://github.com/apache/spark/pull/28501#discussion_r425569841
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -595,8 +601,73 @@ class Analyzer(
}
}
-// This require transformUp to replace grouping()/grouping_id() in
resolved Filter/Sort
-def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
+private def tryResolveHavingCondition(
+a: UnresolvedHaving, havingCond: Expression, agg: LogicalPlan):
LogicalPlan = {
Review comment:
Thanks, done in 3b48e38.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28501: [SPARK-31663][SQL] Grouping sets with having clause returns the wrong result
SparkQA commented on pull request #28501: URL: https://github.com/apache/spark/pull/28501#issuecomment-629031174 **[Test build #122648 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122648/testReport)** for PR 28501 at commit [`3b48e38`](https://github.com/apache/spark/commit/3b48e38d72e35e24d5f47b0a1d36f27f876455d1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #25870: [SPARK-27936][K8S] Support python deps
SparkQA commented on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629028801 **[Test build #122647 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122647/testReport)** for PR 25870 at commit [`a9f055b`](https://github.com/apache/spark/commit/a9f055b4642d487b724a628c4d6f682c71fd2d2c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #25870: [SPARK-27936][K8S] Support python deps
dongjoon-hyun commented on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629027123 The dependency issue is fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #25870: [SPARK-27936][K8S] Support python deps
dongjoon-hyun commented on pull request #25870: URL: https://github.com/apache/spark/pull/25870#issuecomment-629027156 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
AmplabJenkins removed a comment on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629022492 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122646/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
SparkQA removed a comment on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629020341 **[Test build #5006 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/5006/testReport)** for PR 28526 at commit [`9f14144`](https://github.com/apache/spark/commit/9f14144d191c8c41d8b3bd9585d78eb7ddae8407). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
SparkQA commented on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629022933 **[Test build #5006 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/5006/testReport)** for PR 28526 at commit [`9f14144`](https://github.com/apache/spark/commit/9f14144d191c8c41d8b3bd9585d78eb7ddae8407). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class AvroDeserializer(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
AmplabJenkins removed a comment on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629022582 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122645/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
AmplabJenkins removed a comment on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629022578 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
SparkQA removed a comment on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629019973 **[Test build #122646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122646/testReport)** for PR 28526 at commit [`9f14144`](https://github.com/apache/spark/commit/9f14144d191c8c41d8b3bd9585d78eb7ddae8407). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
SparkQA removed a comment on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629012577 **[Test build #122645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122645/testReport)** for PR 28537 at commit [`31d032e`](https://github.com/apache/spark/commit/31d032e8a8525a1527c666b9064d872e1f087460). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
AmplabJenkins removed a comment on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629022488 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
SparkQA commented on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629022481 **[Test build #122646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122646/testReport)** for PR 28526 at commit [`9f14144`](https://github.com/apache/spark/commit/9f14144d191c8c41d8b3bd9585d78eb7ddae8407). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class AvroDeserializer(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
AmplabJenkins commented on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629022488 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
SparkQA commented on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629022542 **[Test build #122645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122645/testReport)** for PR 28537 at commit [`31d032e`](https://github.com/apache/spark/commit/31d032e8a8525a1527c666b9064d872e1f087460). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
AmplabJenkins commented on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629022578 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
HyukjinKwon edited a comment on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629021118 FYI, the issue seems not completely fixed yet due to the cached `.m2/repository`, see also https://issues.apache.org/jira/browse/SPARK-31693?focusedCommentId=17107856=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17107856 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
HyukjinKwon commented on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629021118 FYI,tThis seems not completely fixed yet due to the cached `.m2/repository`, see also https://issues.apache.org/jira/browse/SPARK-31693?focusedCommentId=17107856=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17107856 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
HeartSaVioR commented on a change in pull request #28517:
URL: https://github.com/apache/spark/pull/28517#discussion_r425561029
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##
@@ -2195,22 +2195,4 @@ class DDLParserSuite extends AnalysisTest {
parsePlan("COMMENT ON TABLE a.b.c IS 'xYz'"),
CommentOnTable(UnresolvedTable(Seq("a", "b", "c")), "xYz"))
}
-
- test("create table - without using") {
Review comment:
Yeah I see. That's one of valid approaches and good idea. Maybe then we
need to remove some tests which depend on the config (two tests for on and off).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
SparkQA commented on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629020394 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
SparkQA commented on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629020341 **[Test build #5006 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/5006/testReport)** for PR 28526 at commit [`9f14144`](https://github.com/apache/spark/commit/9f14144d191c8c41d8b3bd9585d78eb7ddae8407). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
AmplabJenkins commented on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629020204 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
AmplabJenkins removed a comment on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629020204 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
SparkQA commented on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629019973 **[Test build #122646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122646/testReport)** for PR 28526 at commit [`9f14144`](https://github.com/apache/spark/commit/9f14144d191c8c41d8b3bd9585d78eb7ddae8407). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
HeartSaVioR commented on a change in pull request #28517: URL: https://github.com/apache/spark/pull/28517#discussion_r425560435 ## File path: docs/sql-ref-syntax-ddl-create-table-datasource.md ## @@ -40,7 +40,7 @@ CREATE TABLE [ IF NOT EXISTS ] table_identifier [ AS select_statement ] ``` -Note that, the clauses between the USING clause and the AS SELECT clause can come in +Note that, the clauses between the OPTIONS clause and the AS SELECT clause can come in Review comment: Ah OK my bad. I found USING shouldn't com in any order so previous statement was correct. Sorry about that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28526: [SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files
cloud-fan commented on pull request #28526: URL: https://github.com/apache/spark/pull/28526#issuecomment-629019709 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28501: [SPARK-31663][SQL] Grouping sets with having clause returns the wrong result
cloud-fan commented on pull request #28501: URL: https://github.com/apache/spark/pull/28501#issuecomment-629019390 It's a long-standing bug, so it doesn't block 2.4.6. I'll see if I can merge it today. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
cloud-fan commented on a change in pull request #28517:
URL: https://github.com/apache/spark/pull/28517#discussion_r425559542
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##
@@ -2195,22 +2195,4 @@ class DDLParserSuite extends AnalysisTest {
parsePlan("COMMENT ON TABLE a.b.c IS 'xYz'"),
CommentOnTable(UnresolvedTable(Seq("a", "b", "c")), "xYz"))
}
-
- test("create table - without using") {
Review comment:
I think he meant to remove
`withSQLConf(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED.key ->
"false")`. We don't need the config anyway, if we unify the syntax.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
cloud-fan commented on a change in pull request #28517: URL: https://github.com/apache/spark/pull/28517#discussion_r425559216 ## File path: docs/sql-ref-syntax-ddl-create-table-datasource.md ## @@ -40,7 +40,7 @@ CREATE TABLE [ IF NOT EXISTS ] table_identifier [ AS select_statement ] ``` -Note that, the clauses between the USING clause and the AS SELECT clause can come in +Note that, the clauses between the OPTIONS clause and the AS SELECT clause can come in Review comment: > between the USING clause and the AS SELECT clause USING and AS SELECT are both not included, so this is still corrected. ## File path: docs/sql-ref-syntax-ddl-create-table-datasource.md ## @@ -40,7 +40,7 @@ CREATE TABLE [ IF NOT EXISTS ] table_identifier [ AS select_statement ] ``` -Note that, the clauses between the USING clause and the AS SELECT clause can come in +Note that, the clauses between the OPTIONS clause and the AS SELECT clause can come in Review comment: > between the USING clause and the AS SELECT clause USING and AS SELECT are both not included, so this is still corrected. We shouldn't change it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
HeartSaVioR commented on a change in pull request #28517: URL: https://github.com/apache/spark/pull/28517#discussion_r425558929 ## File path: docs/sql-ref-syntax-ddl-create-table-datasource.md ## @@ -40,7 +40,7 @@ CREATE TABLE [ IF NOT EXISTS ] table_identifier [ AS select_statement ] ``` -Note that, the clauses between the USING clause and the AS SELECT clause can come in +Note that, the clauses between the OPTIONS clause and the AS SELECT clause can come in Review comment: between is more likely be inclusive, otherwise previous statement was also wrong. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28531: [SPARK-31712][SQL][TESTS] Check casting timestamps before the epoch to Byte/Short/Int/Long types
cloud-fan closed pull request #28531: URL: https://github.com/apache/spark/pull/28531 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
HeartSaVioR commented on a change in pull request #28517:
URL: https://github.com/apache/spark/pull/28517#discussion_r425558612
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##
@@ -2195,22 +2195,4 @@ class DDLParserSuite extends AnalysisTest {
parsePlan("COMMENT ON TABLE a.b.c IS 'xYz'"),
CommentOnTable(UnresolvedTable(Seq("a", "b", "c")), "xYz"))
}
-
- test("create table - without using") {
Review comment:
Sorry but could you please elaborate? Removing config is what this patch
proposed, and it's a part of revert.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28531: [SPARK-31712][SQL][TESTS] Check casting timestamps before the epoch to Byte/Short/Int/Long types
cloud-fan commented on pull request #28531: URL: https://github.com/apache/spark/pull/28531#issuecomment-629018056 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28535: add jvm opt -Xms to reduce FGC
HyukjinKwon commented on pull request #28535: URL: https://github.com/apache/spark/pull/28535#issuecomment-629014875 @oikomi, please keep the PR description and file a JIRA. Also, I think you can specify Xms via `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions` already. Let's don't assume all users want to set the minimum memory. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
AmplabJenkins commented on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629012794 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
AmplabJenkins removed a comment on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629012794 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
SparkQA commented on pull request #28537: URL: https://github.com/apache/spark/pull/28537#issuecomment-629012577 **[Test build #122645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122645/testReport)** for PR 28537 at commit [`31d032e`](https://github.com/apache/spark/commit/31d032e8a8525a1527c666b9064d872e1f087460). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn opened a new pull request #28537: [SPARK-31715][SQL][TEST] Fix flaky SparkSQLEnvSuite that sometimes varies single derby instance standard
yaooqinn opened a new pull request #28537: URL: https://github.com/apache/spark/pull/28537 ### What changes were proposed in this pull request? https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122622/testReport/junit/org.apache.spark.sql.hive.thriftserver/SparkSQLEnvSuite/SPARK_29604_external_listeners_should_be_initialized_with_Spark_classloader/history/?start=25 According to the test report history of SparkSQLEnvSuite，this test fails frequently which is caused by single derby restriction. ```java Caused by: sbt.ForkMain$ForkError: org.apache.derby.iapi.error.StandardException: Another instance of Derby may have already booted the database /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/metastore_db. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.getJBMSLockOnDB(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source) at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source) at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source) at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source) at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source) ... 138 more ``` This PR adds a separate directory to locate the metastore_db for this test which runs in a dedicated JVM. Besides, diable the UI for the potential race on `spark.ui.port` which may also let the test case become flaky. ### Why are the changes needed? test fix ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? SparkSQLEnvSuite itself. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28536: [WIP][SPARK-XXX][SQL] Use a fallback version in HiveExternalCatalogVersionsSuite
AmplabJenkins removed a comment on pull request #28536: URL: https://github.com/apache/spark/pull/28536#issuecomment-629009346 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27306/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28536: [WIP][SPARK-XXX][SQL] Use a fallback version in HiveExternalCatalogVersionsSuite
AmplabJenkins commented on pull request #28536: URL: https://github.com/apache/spark/pull/28536#issuecomment-629009343 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28536: [WIP][SPARK-XXX][SQL] Use a fallback version in HiveExternalCatalogVersionsSuite
AmplabJenkins removed a comment on pull request #28536: URL: https://github.com/apache/spark/pull/28536#issuecomment-629009343 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28536: [WIP][SPARK-XXX][SQL] Use a fallback version in HiveExternalCatalogVersionsSuite
SparkQA commented on pull request #28536: URL: https://github.com/apache/spark/pull/28536#issuecomment-629009075 **[Test build #122644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122644/testReport)** for PR 28536 at commit [`5505ab1`](https://github.com/apache/spark/commit/5505ab1c0d7890750f4a5bd4888befb4154450d6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28222: SPARK-31447 Fix issue in ExtractIntervalPart expression
yaooqinn commented on pull request #28222: URL: https://github.com/apache/spark/pull/28222#issuecomment-629008491 Hi @sathyaprakashg This could be an API change and may cause performance regression for this operator. IMHO, we may need much stronger evidence to make the change, e.g. SQL standard, behaviors for `timestamp - timestamp` from other modern DBMS systems as much as possible. Otherwise, returning only micros part may be spark-specific but efficient and causes no ambiguity. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28536: [WIP][SPARK-XXX][SQL] Use a fallback version in HiveExternalCatalogVersionsSuite
AmplabJenkins removed a comment on pull request #28536: URL: https://github.com/apache/spark/pull/28536#issuecomment-629007660 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28531: [SPARK-31712][SQL][TESTS] Check casting timestamps before the epoch to Byte/Short/Int/Long types
AmplabJenkins removed a comment on pull request #28531: URL: https://github.com/apache/spark/pull/28531#issuecomment-629007220 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28536: [WIP][SPARK-XXX][SQL] Use a fallback version in HiveExternalCatalogVersionsSuite
AmplabJenkins commented on pull request #28536: URL: https://github.com/apache/spark/pull/28536#issuecomment-629007660 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28531: [SPARK-31712][SQL][TESTS] Check casting timestamps before the epoch to Byte/Short/Int/Long types
AmplabJenkins commented on pull request #28531: URL: https://github.com/apache/spark/pull/28531#issuecomment-629007220 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28536: [WIP][SPARK-XXX][SQL] Use a fallback version in HiveExternalCatalogVersionsSuite
SparkQA commented on pull request #28536: URL: https://github.com/apache/spark/pull/28536#issuecomment-629007295 **[Test build #122643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122643/testReport)** for PR 28536 at commit [`6d44e9e`](https://github.com/apache/spark/commit/6d44e9e1e216a9135ab3e823358cc7d05788f33c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #28536: [WIP][SPARK-XXX][SQL] Use a fallback version in HiveExternalCatalogVersionsSuite
dongjoon-hyun opened a new pull request #28536: URL: https://github.com/apache/spark/pull/28536 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28531: [SPARK-31712][SQL][TESTS] Check casting timestamps before the epoch to Byte/Short/Int/Long types
SparkQA removed a comment on pull request #28531: URL: https://github.com/apache/spark/pull/28531#issuecomment-628901532 **[Test build #122634 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122634/testReport)** for PR 28531 at commit [`d6f1ec6`](https://github.com/apache/spark/commit/d6f1ec6ee9429abac6d52b0426706fba627fbaa9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28531: [SPARK-31712][SQL][TESTS] Check casting timestamps before the epoch to Byte/Short/Int/Long types
SparkQA commented on pull request #28531: URL: https://github.com/apache/spark/pull/28531#issuecomment-629006662 **[Test build #122634 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122634/testReport)** for PR 28531 at commit [`d6f1ec6`](https://github.com/apache/spark/commit/d6f1ec6ee9429abac6d52b0426706fba627fbaa9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
Ngone51 commented on a change in pull request #28517:
URL: https://github.com/apache/spark/pull/28517#discussion_r425546566
##
File path: docs/sql-ref-syntax-ddl-create-table-datasource.md
##
@@ -40,7 +40,7 @@ CREATE TABLE [ IF NOT EXISTS ] table_identifier
[ AS select_statement ]
```
-Note that, the clauses between the USING clause and the AS SELECT clause can
come in
+Note that, the clauses between the OPTIONS clause and the AS SELECT clause can
come in
Review comment:
Should still be `USING` since `OPTIONS` itself can also come in any
order?
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##
@@ -2195,22 +2195,4 @@ class DDLParserSuite extends AnalysisTest {
parsePlan("COMMENT ON TABLE a.b.c IS 'xYz'"),
CommentOnTable(UnresolvedTable(Seq("a", "b", "c")), "xYz"))
}
-
- test("create table - without using") {
Review comment:
So, just remove the conf after revert?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #28501: [SPARK-31663][SQL] Grouping sets with having clause returns the wrong result
xuanyuanking commented on pull request #28501: URL: https://github.com/apache/spark/pull/28501#issuecomment-629004304 @holdenk Thanks for notifying, I'll address all the comments today. Yep, if it can be merged before cutting 2.4.6, let's put it in 2.4.7. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28496: [SPARK-31620][SQL] Fix reference binding failure in case of an final agg contains subquery
AmplabJenkins commented on pull request #28496: URL: https://github.com/apache/spark/pull/28496#issuecomment-629003873 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28496: [SPARK-31620][SQL] Fix reference binding failure in case of an final agg contains subquery
AmplabJenkins removed a comment on pull request #28496: URL: https://github.com/apache/spark/pull/28496#issuecomment-629003873 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28496: [SPARK-31620][SQL] Fix reference binding failure in case of an final agg contains subquery
SparkQA commented on pull request #28496: URL: https://github.com/apache/spark/pull/28496#issuecomment-629003544 **[Test build #122642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122642/testReport)** for PR 28496 at commit [`493157a`](https://github.com/apache/spark/commit/493157a3b97616d221ec2b5ddf1a21cdf9a1a3f4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28496: [SPARK-31620][SQL] Fix reference binding failure in case of an final agg contains subquery
AmplabJenkins removed a comment on pull request #28496: URL: https://github.com/apache/spark/pull/28496#issuecomment-628738076 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122619/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #28496: [SPARK-31620][SQL] Fix reference binding failure in case of an final agg contains subquery
Ngone51 commented on pull request #28496: URL: https://github.com/apache/spark/pull/28496#issuecomment-629001780 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #28517: [SPARK-31707][SQL] Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
HeartSaVioR commented on a change in pull request #28517:
URL: https://github.com/apache/spark/pull/28517#discussion_r425540192
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##
@@ -2195,22 +2195,4 @@ class DDLParserSuite extends AnalysisTest {
parsePlan("COMMENT ON TABLE a.b.c IS 'xYz'"),
CommentOnTable(UnresolvedTable(Seq("a", "b", "c")), "xYz"))
}
-
- test("create table - without using") {
Review comment:
It cannot be simply changed to "ignore" instead of "test" as we're
removing the config. I'll comment out instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data
SparkQA commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-628995389 **[Test build #122641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122641/testReport)** for PR 28525 at commit [`bb15a4d`](https://github.com/apache/spark/commit/bb15a4dab44fdba03cf8fcab06c591f16ca5c4e6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
