[GitHub] [spark] SparkQA commented on pull request #34494: [SPARK-36924][SQL] CAST between ANSI intervals and IntegralType
SparkQA commented on pull request #34494: URL: https://github.com/apache/spark/pull/34494#issuecomment-969972906 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49739/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on pull request #34602: [SPARK-37328][SQL] Fix bug that OptimizeSkewedJoin may not work after it was moved from queryStageOptimizerRules to queryStagePrepar
cloud-fan edited a comment on pull request #34602: URL: https://github.com/apache/spark/pull/34602#issuecomment-969968796 `OptimizeSkewedJoin` is supposed to only handle materialized shuffle stages, or did I miss something? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34602: [SPARK-37328][SQL] Fix bug that OptimizeSkewedJoin may not work after it was moved from queryStageOptimizerRules to queryStagePreparationRu
cloud-fan commented on pull request #34602: URL: https://github.com/apache/spark/pull/34602#issuecomment-969968796 `OptimizeSkewedJoin` is supposed to only handle materialized states, or did I miss something? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
SparkQA removed a comment on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969967369 **[Test build #145272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145272/testReport)** for PR 34466 at commit [`8787ac3`](https://github.com/apache/spark/commit/8787ac398098d5960dc93a295de1ff4fde6a157a). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
AmplabJenkins removed a comment on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969967767 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145272/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
AmplabJenkins commented on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969967767 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145272/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
SparkQA commented on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969967732 **[Test build #145272 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145272/testReport)** for PR 34466 at commit [`8787ac3`](https://github.com/apache/spark/commit/8787ac398098d5960dc93a295de1ff4fde6a157a). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34603: [SPARK-37702][SQL][FOLLOWUP] Store referred temp functions for CacheTableAsSelect
cloud-fan commented on a change in pull request #34603:
URL: https://github.com/apache/spark/pull/34603#discussion_r749986111
##
File path: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala
##
@@ -1643,4 +1643,12 @@ class CachedTableSuite extends QueryTest with
SQLTestUtils
}
}
}
+
+ test("SPARK-37702: cache table with temporary function ") {
+spark.udf.register("udf", (id: Int) => id + 1)
Review comment:
let's wrap with `withUserDefinedFunction`, so that we can unregister the
function at the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
SparkQA commented on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969967369 **[Test build #145272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145272/testReport)** for PR 34466 at commit [`8787ac3`](https://github.com/apache/spark/commit/8787ac398098d5960dc93a295de1ff4fde6a157a). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #33588: [SPARK-36346][SQL] Support TimestampNTZ type in Orc file source
beliefer commented on pull request #33588: URL: https://github.com/apache/spark/pull/33588#issuecomment-969963221 ping @gengliangwang @cloud-fan @MaxGekk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
SparkQA removed a comment on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969958804 **[Test build #145271 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145271/testReport)** for PR 34466 at commit [`ddbcaf5`](https://github.com/apache/spark/commit/ddbcaf5acf64ab01d7e9effaea5aec60ec4d981c). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
AmplabJenkins removed a comment on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969959134 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145271/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
MaxGekk closed pull request #34600: URL: https://github.com/apache/spark/pull/34600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
AmplabJenkins commented on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969959134 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145271/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
SparkQA commented on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969959113 **[Test build #145271 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145271/testReport)** for PR 34466 at commit [`ddbcaf5`](https://github.com/apache/spark/commit/ddbcaf5acf64ab01d7e9effaea5aec60ec4d981c). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
cloud-fan commented on a change in pull request #34497:
URL: https://github.com/apache/spark/pull/34497#discussion_r749971719
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/SupportsCatalogOptionsSuite.scala
##
@@ -271,6 +276,54 @@ class SupportsCatalogOptionsSuite extends QueryTest with
SharedSparkSession with
}
}
+ test("mock time travel test") {
+sql(s"create table $catalogName.tSnapshot123456789 (id bigint) using
$format")
Review comment:
let's wrap the test with `withTables`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34600: [SPARK-37332][SQL] Allow ANSI intervals in `ALTER TABLE .. ADD COLUMNS`
MaxGekk commented on pull request #34600: URL: https://github.com/apache/spark/pull/34600#issuecomment-969958940 Merging to master. Thank you, @sarutak and @cloud-fan for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
SparkQA commented on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969958804 **[Test build #145271 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145271/testReport)** for PR 34466 at commit [`ddbcaf5`](https://github.com/apache/spark/commit/ddbcaf5acf64ab01d7e9effaea5aec60ec4d981c). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] c21 commented on pull request #34612: [SPARK-37341][SQL] Avoid unnecessary buffer and copy in full outer sort merge join
c21 commented on pull request #34612: URL: https://github.com/apache/spark/pull/34612#issuecomment-969958618 cc @cloud-fan could you help take a look when you have time? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
AmplabJenkins removed a comment on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969956719 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34614: [SPARK-37277][PYTHON][SQL] Support DayTimeIntervalType in createDataFrame, collect and Python UDF
AmplabJenkins removed a comment on pull request #34614: URL: https://github.com/apache/spark/pull/34614#issuecomment-969956726 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49737/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
cloud-fan commented on a change in pull request #34497:
URL: https://github.com/apache/spark/pull/34497#discussion_r749969978
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##
@@ -2371,4 +2371,8 @@ object QueryCompilationErrors {
new AnalysisException(
s"Invalid view text: $viewText. The view $tableName may have been
tampered with")
}
+
+ def invalidTimeTravelSpecError(errorMessage: String): Throwable = {
Review comment:
Seems like this method does not need a parameter. We can write the error
message here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
AmplabJenkins removed a comment on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-969956722 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145262/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34614: [SPARK-37277][PYTHON][SQL] Support DayTimeIntervalType in createDataFrame, collect and Python UDF
AmplabJenkins commented on pull request #34614: URL: https://github.com/apache/spark/pull/34614#issuecomment-969956726 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49737/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
AmplabJenkins commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969956719 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
AmplabJenkins commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-969956722 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145262/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
cloud-fan commented on a change in pull request #34497:
URL: https://github.com/apache/spark/pull/34497#discussion_r749969426
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##
@@ -1257,7 +1257,27 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with
SQLConfHelper with Logg
*/
override def visitTableName(ctx: TableNameContext): LogicalPlan =
withOrigin(ctx) {
val tableId = visitMultipartIdentifier(ctx.multipartIdentifier)
-val table = mayApplyAliasPlan(ctx.tableAlias, UnresolvedRelation(tableId))
+val version: Option[String] =
+ if (ctx.temporalClause != null && ctx.temporalClause.version != null) {
+if (ctx.temporalClause.version.getText.forall(Character.isDigit)) {
+ Some(ctx.temporalClause.version.getText)
+} else {
+ Some(string(ctx.temporalClause.version))
+}
+ } else {
+None
+ }
+val timeTravel = if (ctx.temporalClause != null) {
+ TimeTravelSpec.create(
+Option(ctx.temporalClause.timestamp).map(string),
+version)
+} else {
+ None
+}
+
+val table = mayApplyAliasPlan(ctx.tableAlias,
+ UnresolvedRelation(tableId,
+timeTravelSpec = timeTravel))
Review comment:
```suggestion
UnresolvedRelation(tableId, timeTravelSpec = timeTravel))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
cloud-fan commented on a change in pull request #34497:
URL: https://github.com/apache/spark/pull/34497#discussion_r749968932
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##
@@ -1257,7 +1257,27 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with
SQLConfHelper with Logg
*/
override def visitTableName(ctx: TableNameContext): LogicalPlan =
withOrigin(ctx) {
val tableId = visitMultipartIdentifier(ctx.multipartIdentifier)
-val table = mayApplyAliasPlan(ctx.tableAlias, UnresolvedRelation(tableId))
+val version: Option[String] =
+ if (ctx.temporalClause != null && ctx.temporalClause.version != null) {
+if (ctx.temporalClause.version.getText.forall(Character.isDigit)) {
Review comment:
can't we check `ctx.temporalClause.version.INTEGER_VALUE != null`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
SparkQA commented on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969949235 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49740/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
cloud-fan commented on a change in pull request #34497:
URL: https://github.com/apache/spark/pull/34497#discussion_r749966383
##
File path:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##
@@ -599,6 +599,11 @@ fromClause
: FROM relation (',' relation)* lateralView* pivotClause?
;
+temporalClause
+: FOR? (SYSTEM_VERSION | VERSION) AS OF version=(INTEGER_VALUE | STRING)
Review comment:
do you mean `((FOR SYSTEM_VERSION) | VERSION)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
SparkQA removed a comment on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969706020 **[Test build #145263 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145263/testReport)** for PR 34575 at commit [`c531300`](https://github.com/apache/spark/commit/c5313007953510175a3275421ce8c040ee9669e3). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
SparkQA commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969947618 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49738/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
SparkQA commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969947242 **[Test build #145263 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145263/testReport)** for PR 34575 at commit [`c531300`](https://github.com/apache/spark/commit/c5313007953510175a3275421ce8c040ee9669e3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
cloud-fan commented on a change in pull request #34497:
URL: https://github.com/apache/spark/pull/34497#discussion_r749966383
##
File path:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##
@@ -599,6 +599,11 @@ fromClause
: FROM relation (',' relation)* lateralView* pivotClause?
;
+temporalClause
+: FOR? (SYSTEM_VERSION | VERSION) AS OF version=(INTEGER_VALUE | STRING)
Review comment:
do you mean `( (FOR SYSTEM_VERSION) | VERSION)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
SparkQA removed a comment on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-969705868 **[Test build #145262 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145262/testReport)** for PR 34596 at commit [`ea47b94`](https://github.com/apache/spark/commit/ea47b9439028585e1f1a383b170150609924d4c7). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source
SparkQA commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-969942801 **[Test build #145262 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145262/testReport)** for PR 34596 at commit [`ea47b94`](https://github.com/apache/spark/commit/ea47b9439028585e1f1a383b170150609924d4c7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34494: [SPARK-36924][SQL] CAST between ANSI intervals and IntegralType
SparkQA commented on pull request #34494: URL: https://github.com/apache/spark/pull/34494#issuecomment-969940491 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49739/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34614: [SPARK-37277][PYTHON][SQL] Support DayTimeIntervalType in createDataFrame, collect and Python UDF
SparkQA commented on pull request #34614: URL: https://github.com/apache/spark/pull/34614#issuecomment-969934263 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49737/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sleep1661 commented on a change in pull request #34578: [SPARK-37300][CORE] TaskSchedulerImpl should ignore task finished eve…
sleep1661 commented on a change in pull request #34578:
URL: https://github.com/apache/spark/pull/34578#discussion_r749952744
##
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##
@@ -871,19 +871,23 @@ private[spark] class TaskSchedulerImpl(
taskSetManager: TaskSetManager,
tid: Long,
taskResult: DirectTaskResult[_]): Unit = synchronized {
-taskSetManager.handleSuccessfulTask(tid, taskResult)
+if (!taskSetManager.taskFinished(tid)) {
+ taskSetManager.handleSuccessfulTask(tid, taskResult)
+}
}
def handleFailedTask(
taskSetManager: TaskSetManager,
tid: Long,
taskState: TaskState,
reason: TaskFailedReason): Unit = synchronized {
-taskSetManager.handleFailedTask(tid, taskState, reason)
-if (!taskSetManager.isZombie && !taskSetManager.someAttemptSucceeded(tid))
{
- // Need to revive offers again now that the task set manager state has
been updated to
- // reflect failed tasks that need to be re-run.
- backend.reviveOffers()
+if (!taskSetManager.taskFinished(tid)) {
+ taskSetManager.handleFailedTask(tid, taskState, reason)
Review comment:
Yes, i agree. Inside state checking will be more safer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34580: [SPARK-37270][SQL] Fix push foldable into CaseWhen branches if elseValue is empty
AmplabJenkins removed a comment on pull request #34580: URL: https://github.com/apache/spark/pull/34580#issuecomment-969913147 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145258/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zinking commented on a change in pull request #18440: [SPARK-21229][SQL] remove QueryPlan.preCanonicalized
zinking commented on a change in pull request #18440: URL: https://github.com/apache/spark/pull/18440#discussion_r749945481 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -138,8 +138,12 @@ case class RowDataSourceScanExec( } // Only care about `relation` and `metadata` when canonicalizing. - override def preCanonicalized: SparkPlan = -copy(rdd = null, outputPartitioning = null, metastoreTableIdentifier = None) + override lazy val canonicalized: SparkPlan = +copy( + output.map(QueryPlan.normalizeExprId(_, output)), + rdd = null, + outputPartitioning = null, + metastoreTableIdentifier = None) Review comment: @cloud-fan is it right to remove table identifiers here, could it mistakenly link two different tables with same schema together? ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala ## @@ -43,7 +44,9 @@ case class LogicalRelation( } // Only care about relation when canonicalizing. - override def preCanonicalized: LogicalPlan = copy(catalogTable = None) + override lazy val canonicalized: LogicalPlan = copy( +output = output.map(QueryPlan.normalizeExprId(_, output)), +catalogTable = None) Review comment: same here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34580: [SPARK-37270][SQL] Fix push foldable into CaseWhen branches if elseValue is empty
AmplabJenkins commented on pull request #34580: URL: https://github.com/apache/spark/pull/34580#issuecomment-969913147 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145258/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
SparkQA removed a comment on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969912533 **[Test build #145270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145270/testReport)** for PR 34466 at commit [`ff44294`](https://github.com/apache/spark/commit/ff4429430448dcd3d1b3c0b086feac79408e1395). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
AmplabJenkins commented on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969912878 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145270/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
SparkQA commented on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969912864 **[Test build #145270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145270/testReport)** for PR 34466 at commit [`ff44294`](https://github.com/apache/spark/commit/ff4429430448dcd3d1b3c0b086feac79408e1395). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
AmplabJenkins removed a comment on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969912878 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145270/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34494: [SPARK-36924][SQL] CAST between ANSI intervals and IntegralType
SparkQA commented on pull request #34494: URL: https://github.com/apache/spark/pull/34494#issuecomment-969912548 **[Test build #145269 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145269/testReport)** for PR 34494 at commit [`686a122`](https://github.com/apache/spark/commit/686a122b3759e849d310ff4cca65fc8389d83a16). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34466: [SPARK-37152][PYTHON] Inline type hints for python/pyspark/context.py
SparkQA commented on pull request #34466: URL: https://github.com/apache/spark/pull/34466#issuecomment-969912533 **[Test build #145270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145270/testReport)** for PR 34466 at commit [`ff44294`](https://github.com/apache/spark/commit/ff4429430448dcd3d1b3c0b086feac79408e1395). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34580: [SPARK-37270][SQL] Fix push foldable into CaseWhen branches if elseValue is empty
SparkQA removed a comment on pull request #34580: URL: https://github.com/apache/spark/pull/34580#issuecomment-969612699 **[Test build #145258 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145258/testReport)** for PR 34580 at commit [`31f5a50`](https://github.com/apache/spark/commit/31f5a507611c551bbd64114690a57823ba86fd4d). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34580: [SPARK-37270][SQL] Fix push foldable into CaseWhen branches if elseValue is empty
SparkQA commented on pull request #34580: URL: https://github.com/apache/spark/pull/34580#issuecomment-969912205 **[Test build #145258 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145258/testReport)** for PR 34580 at commit [`31f5a50`](https://github.com/apache/spark/commit/31f5a507611c551bbd64114690a57823ba86fd4d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34614: [SPARK-37277][PYTHON][SQL] Support DayTimeIntervalType in createDataFrame, collect and Python UDF
AmplabJenkins removed a comment on pull request #34614: URL: https://github.com/apache/spark/pull/34614#issuecomment-969906630 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49735/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
AmplabJenkins removed a comment on pull request #34497: URL: https://github.com/apache/spark/pull/34497#issuecomment-969906629 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49736/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
AmplabJenkins commented on pull request #34497: URL: https://github.com/apache/spark/pull/34497#issuecomment-969906629 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49736/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down
AmplabJenkins commented on pull request #34593: URL: https://github.com/apache/spark/pull/34593#issuecomment-969906661 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34614: [SPARK-37277][PYTHON][SQL] Support DayTimeIntervalType in createDataFrame, collect and Python UDF
AmplabJenkins commented on pull request #34614: URL: https://github.com/apache/spark/pull/34614#issuecomment-969906630 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49735/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34494: [SPARK-36924][SQL] CAST between ANSI intervals and IntegralType
Peng-Lei commented on a change in pull request #34494:
URL: https://github.com/apache/spark/pull/34494#discussion_r749941998
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##
@@ -1253,4 +1255,50 @@ object IntervalUtils {
}
intervalString
}
+
+ def integralToYearMonthInterval(v: Long, endField: Byte): Int = {
+if (v != v.toInt) {
+ throw QueryExecutionErrors.castingCauseOverflowError(v,
YM(endField).catalogString)
+}
+endField match {
+ case YEAR =>
+try {
+ Math.multiplyExact(v.toInt, MONTHS_PER_YEAR)
+} catch {
+ case _: ArithmeticException =>
+throw QueryExecutionErrors.castingCauseOverflowError(v,
YM(endField).catalogString)
+}
+ case MONTH => v.toInt
+}
+ }
+
+ def yearMonthIntervalToInt(v: Int, endFiled: Byte): Int = {
+endFiled match {
+ case YEAR => v / MONTHS_PER_YEAR
+ case MONTH => v
+}
+ }
+
+ def integralToDayTimeInterval(v: Long, endField: Byte): Long = {
+try {
+ endField match {
+case DAY => Math.multiplyExact(v, MICROS_PER_DAY)
+case HOUR => Math.multiplyExact(v, MICROS_PER_HOUR)
+case MINUTE => Math.multiplyExact(v, MICROS_PER_MINUTE)
+case SECOND => Math.multiplyExact(v, MICROS_PER_SECOND)
+ }
+} catch {
+ case _: ArithmeticException =>
+throw QueryExecutionErrors.castingCauseOverflowError(v,
DT(endField).catalogString)
+}
+ }
+
+ def dayTimeIntervalToLong(v: Long, endFiled: Byte): Long = {
+endFiled match {
+ case DAY => v / MICROS_PER_DAY
+ case HOUR => v / MICROS_PER_HOUR
+ case MINUTE => v / MICROS_PER_MINUTE
+ case SECOND => v / MICROS_PER_SECOND
+}
+ }
Review comment:
I thought about it. Finally, as you said, the corresponding convert
function is defined even if it will be used infrequently. So the codegen just
call the corresponding convert function that is better match and the overflow
check is performed only when necessary for the corresponding convert function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
SparkQA commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969903285 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49738/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34494: [SPARK-36924][SQL] CAST between ANSI intervals and IntegralType
Peng-Lei commented on a change in pull request #34494:
URL: https://github.com/apache/spark/pull/34494#discussion_r749939106
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##
@@ -580,4 +584,329 @@ class CastSuite extends CastSuiteBase {
test("SPARK-36286: invalid string cast to timestamp") {
checkEvaluation(cast(Literal("2015-03-18T"), TimestampType), null)
}
+
+ test("SPARK-36924: Cast DayTimeIntervalType to IntegralType") {
+DataTypeTestUtils.dayTimeIntervalTypes.foreach { dt =>
+ val v1 = Literal.create(Duration.ZERO, dt)
+ checkEvaluation(cast(v1, ByteType), 0.toByte)
+ checkEvaluation(cast(v1, ShortType), 0.toShort)
+ checkEvaluation(cast(v1, IntegerType), 0)
+ checkEvaluation(cast(v1, LongType), 0L)
+
+ val num = SECONDS_PER_DAY + SECONDS_PER_HOUR + SECONDS_PER_MINUTE + 1
+ val v2 = Literal.create(Duration.ofSeconds(num), dt)
+ dt.endField match {
+case DAY =>
+ checkEvaluation(cast(v2, ByteType), 1.toByte)
+ checkEvaluation(cast(v2, ShortType), 1.toShort)
+ checkEvaluation(cast(v2, IntegerType), 1)
+ checkEvaluation(cast(v2, LongType), 1L)
+case HOUR =>
+ checkEvaluation(cast(v2, ByteType), 25.toByte)
+ checkEvaluation(cast(v2, ShortType), 25.toShort)
+ checkEvaluation(cast(v2, IntegerType), 25)
+ checkEvaluation(cast(v2, LongType), 25L)
+case MINUTE =>
+ val e = intercept[ArithmeticException] {
+cast(v2, ByteType).eval()
Review comment:
done
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##
@@ -580,4 +584,329 @@ class CastSuite extends CastSuiteBase {
test("SPARK-36286: invalid string cast to timestamp") {
checkEvaluation(cast(Literal("2015-03-18T"), TimestampType), null)
}
+
+ test("SPARK-36924: Cast DayTimeIntervalType to IntegralType") {
+DataTypeTestUtils.dayTimeIntervalTypes.foreach { dt =>
+ val v1 = Literal.create(Duration.ZERO, dt)
+ checkEvaluation(cast(v1, ByteType), 0.toByte)
+ checkEvaluation(cast(v1, ShortType), 0.toShort)
+ checkEvaluation(cast(v1, IntegerType), 0)
+ checkEvaluation(cast(v1, LongType), 0L)
+
+ val num = SECONDS_PER_DAY + SECONDS_PER_HOUR + SECONDS_PER_MINUTE + 1
+ val v2 = Literal.create(Duration.ofSeconds(num), dt)
+ dt.endField match {
+case DAY =>
+ checkEvaluation(cast(v2, ByteType), 1.toByte)
+ checkEvaluation(cast(v2, ShortType), 1.toShort)
+ checkEvaluation(cast(v2, IntegerType), 1)
+ checkEvaluation(cast(v2, LongType), 1L)
+case HOUR =>
+ checkEvaluation(cast(v2, ByteType), 25.toByte)
+ checkEvaluation(cast(v2, ShortType), 25.toShort)
+ checkEvaluation(cast(v2, IntegerType), 25)
+ checkEvaluation(cast(v2, LongType), 25L)
+case MINUTE =>
+ val e = intercept[ArithmeticException] {
+cast(v2, ByteType).eval()
+ }.getMessage
+ assert(e.contains(s"Casting 9006000 to tinyint causes overflow"))
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
SparkQA commented on pull request #34497: URL: https://github.com/apache/spark/pull/34497#issuecomment-969900974 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49736/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34614: [SPARK-37277][PYTHON][SQL] Support DayTimeIntervalType in createDataFrame, collect and Python UDF
SparkQA commented on pull request #34614: URL: https://github.com/apache/spark/pull/34614#issuecomment-969900347 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49737/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34614: [SPARK-37277][PYTHON][SQL] Support DayTimeIntervalType in createDataFrame, collect and Python UDF
SparkQA commented on pull request #34614: URL: https://github.com/apache/spark/pull/34614#issuecomment-969892300 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49735/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
AmplabJenkins removed a comment on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969879005 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145256/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
AmplabJenkins commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969879005 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145256/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
AmplabJenkins removed a comment on pull request #34497: URL: https://github.com/apache/spark/pull/34497#issuecomment-969878553 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145257/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #34613: [SPARK-37342][BUILD] Upgrade Apache Arrow to 6.0.0
HyukjinKwon edited a comment on pull request #34613: URL: https://github.com/apache/spark/pull/34613#issuecomment-969740036 oh you can do, instead: ```bash pip install -r dev/requirements.txt pip install pyarrow==6.0.0 python/run-tests --modules pyspark-sql ``` that would verify all the things -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
AmplabJenkins commented on pull request #34497: URL: https://github.com/apache/spark/pull/34497#issuecomment-969878553 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145257/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
SparkQA removed a comment on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969503205 **[Test build #145256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145256/testReport)** for PR 34575 at commit [`73593c5`](https://github.com/apache/spark/commit/73593c5a9a26889709b57e938a3f6efca14ab1fb). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #34580: [SPARK-37270][SQL] Fix push foldable into CaseWhen branches if elseValue is empty
wangyum commented on a change in pull request #34580:
URL: https://github.com/apache/spark/pull/34580#discussion_r749913907
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##
@@ -678,10 +684,11 @@ object PushFoldableIntoBranches extends Rule[LogicalPlan]
with PredicateHelper {
falseValue = u.withNewChildren(Array(falseValue)))
case u @ UnaryExpression(c @ CaseWhen(branches, elseValue))
- if supportedUnaryExpression(u) &&
atMostOneUnfoldable(branches.map(_._2) ++ elseValue) =>
+ if supportedUnaryExpression(u) &&
atMostOneUnfoldable(branches.map(_._2) :+
+elseValue.getOrElse(Literal(null, c.dataType))) =>
Review comment:
Yes. It can push down more filters. For example:
```scala
spark.sql("CREATE TABLE t1(a int, b int) using parquet")
spark.sql("SELECT * FROM t1 WHERE CASE WHEN a > 10 THEN b END = 20").explain
```
Pushdown:
```
== Physical Plan ==
*(1) Filter (((isnotnull(a#0) AND isnotnull(b#1)) AND (a#0 > 10)) AND (b#1 =
20))
+- *(1) ColumnarToRow
+- FileScan parquet default.t1[a#0,b#1] Batched: true, DataFilters:
[isnotnull(a#0), isnotnull(b#1), (a#0 > 10), (b#1 = 20)], Format: Parquet,
Location: InMemoryFileIndex(1
paths)[file:/Users/yumwang/spark/SPARK-31890/spark-warehouse/org.apache.spark...,
PartitionFilters: [], PushedFilters: [IsNotNull(a), IsNotNull(b),
GreaterThan(a,10), EqualTo(b,20)], ReadSchema: struct
```
Do not pushdown:
```
== Physical Plan ==
*(1) Filter (CASE WHEN (a#0 > 10) THEN b#1 END = 20)
+- *(1) ColumnarToRow
+- FileScan parquet default.t1[a#0,b#1] Batched: true, DataFilters:
[(CASE WHEN (a#0 > 10) THEN b#1 END = 20)], Format: Parquet, Location:
InMemoryFileIndex(1
paths)[file:/Users/yumwang/spark/SPARK-31890/spark-warehouse/org.apache.spark...,
PartitionFilters: [], PushedFilters: [], ReadSchema: struct
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
SparkQA commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969878182 **[Test build #145256 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145256/testReport)** for PR 34575 at commit [`73593c5`](https://github.com/apache/spark/commit/73593c5a9a26889709b57e938a3f6efca14ab1fb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
SparkQA removed a comment on pull request #34497: URL: https://github.com/apache/spark/pull/34497#issuecomment-969503754 **[Test build #145257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145257/testReport)** for PR 34497 at commit [`398950f`](https://github.com/apache/spark/commit/398950f9a0b8cf77fae22167bfb80348a5a6b79e). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34608: [SPARK-37337] Improve the API of Spark DataFrame to pandas-on-Spark DataFrame conversion
AmplabJenkins removed a comment on pull request #34608: URL: https://github.com/apache/spark/pull/34608#issuecomment-969872541 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49734/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
SparkQA commented on pull request #34497: URL: https://github.com/apache/spark/pull/34497#issuecomment-969877695 **[Test build #145257 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145257/testReport)** for PR 34497 at commit [`398950f`](https://github.com/apache/spark/commit/398950f9a0b8cf77fae22167bfb80348a5a6b79e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
SparkQA commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969875498 **[Test build #145268 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145268/testReport)** for PR 34575 at commit [`bd28eb7`](https://github.com/apache/spark/commit/bd28eb7075c3fbd8f003ad7ade1f596795e4ef2a). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34614: [SPARK-37277][PYTHON][SQL] Support DayTimeIntervalType in createDataFrame, collect and Python UDF
SparkQA commented on pull request #34614: URL: https://github.com/apache/spark/pull/34614#issuecomment-969875360 **[Test build #145267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145267/testReport)** for PR 34614 at commit [`4771b6b`](https://github.com/apache/spark/commit/4771b6b1cbe8eda827eb7775eebb02d26f5cfeec). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34612: [SPARK-37341][SQL] Avoid unnecessary buffer and copy in full outer sort merge join
AmplabJenkins removed a comment on pull request #34612: URL: https://github.com/apache/spark/pull/34612#issuecomment-969872542 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145255/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34608: [SPARK-37337] Improve the API of Spark DataFrame to pandas-on-Spark DataFrame conversion
AmplabJenkins commented on pull request #34608: URL: https://github.com/apache/spark/pull/34608#issuecomment-969872541 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49734/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34612: [SPARK-37341][SQL] Avoid unnecessary buffer and copy in full outer sort merge join
AmplabJenkins commented on pull request #34612: URL: https://github.com/apache/spark/pull/34612#issuecomment-969872542 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145255/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34614: [SPARK-37277][PYTHON][SQL] Support DayTimeIntervalType in createDataFrame, collect and Python UDF
SparkQA commented on pull request #34614: URL: https://github.com/apache/spark/pull/34614#issuecomment-969869936 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49735/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
SparkQA commented on pull request #34497: URL: https://github.com/apache/spark/pull/34497#issuecomment-969868907 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49736/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34608: [SPARK-37337] Improve the API of Spark DataFrame to pandas-on-Spark DataFrame conversion
SparkQA commented on pull request #34608: URL: https://github.com/apache/spark/pull/34608#issuecomment-969868456 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49734/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34612: [SPARK-37341][SQL] Avoid unnecessary buffer and copy in full outer sort merge join
SparkQA removed a comment on pull request #34612: URL: https://github.com/apache/spark/pull/34612#issuecomment-969502947 **[Test build #145255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145255/testReport)** for PR 34612 at commit [`979dfbc`](https://github.com/apache/spark/commit/979dfbc21bd3fa3b057958b2c7d87fcbfc220058). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34612: [SPARK-37341][SQL] Avoid unnecessary buffer and copy in full outer sort merge join
SparkQA commented on pull request #34612: URL: https://github.com/apache/spark/pull/34612#issuecomment-969867892 **[Test build #145255 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145255/testReport)** for PR 34612 at commit [`979dfbc`](https://github.com/apache/spark/commit/979dfbc21bd3fa3b057958b2c7d87fcbfc220058). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34580: [SPARK-37270][SQL] Fix push foldable into CaseWhen branches if elseValue is empty
cloud-fan commented on a change in pull request #34580:
URL: https://github.com/apache/spark/pull/34580#discussion_r749900372
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##
@@ -678,10 +684,11 @@ object PushFoldableIntoBranches extends Rule[LogicalPlan]
with PredicateHelper {
falseValue = u.withNewChildren(Array(falseValue)))
case u @ UnaryExpression(c @ CaseWhen(branches, elseValue))
- if supportedUnaryExpression(u) &&
atMostOneUnfoldable(branches.map(_._2) ++ elseValue) =>
+ if supportedUnaryExpression(u) &&
atMostOneUnfoldable(branches.map(_._2) :+
+elseValue.getOrElse(Literal(null, c.dataType))) =>
Review comment:
If the CaseWhen has only one branch and has no else branch, we can still
pushdown?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34610: [SPARK-34332][SQL][TEST] Unify v1 and v2 ALTER TABLE .. SET LOCATION tests
cloud-fan commented on a change in pull request #34610:
URL: https://github.com/apache/spark/pull/34610#discussion_r749898731
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/AlterNamespaceSetLocationSuite.scala
##
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution.command
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.command.v1
+
+/**
+ * The class contains tests for the `ALTER NAMESPACE ... SET LOCATION` command
to check
+ * V1 Hive external table catalog.
+ */
+class AlterNamespaceSetLocationSuite extends
v1.AlterNamespaceSetLocationSuiteBase
+with CommandSuiteBase {
Review comment:
shall we make the base trait extends `CommandSuiteBase`? seems every
leaf suite extends it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34610: [SPARK-34332][SQL][TEST] Unify v1 and v2 ALTER TABLE .. SET LOCATION tests
cloud-fan commented on a change in pull request #34610:
URL: https://github.com/apache/spark/pull/34610#discussion_r749898296
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterNamespaceSetLocationSuite.scala
##
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `ALTER NAMESPACE ... SET
LOCATION` command that
+ * checks V1 table catalogs. The tests that cannot run for all V1 catalogs are
located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog:
+ *
`org.apache.spark.sql.execution.command.v1.AlterNamespaceSetLocationSuite`
+ * - V1 Hive External catalog:
+ *
`org.apache.spark.sql.hive.execution.command.AlterNamespaceSetLocationSuite`
+ */
+trait AlterNamespaceSetLocationSuiteBase extends
command.AlterNamespaceSetLocationSuiteBase
+with command.TestsV1AndV2Commands {
+ override def notFoundMsgPrefix: String = "Database"
+
+ test ("Empty location string") {
+val ns = "db1"
+withNamespace(ns) {
+ sql(s"CREATE NAMESPACE $catalog.$ns")
+ val message = intercept[IllegalArgumentException] {
+sql(s"ALTER DATABASE $catalog.$ns SET LOCATION ''")
+ }.getMessage
+ assert(message.contains("Can not create a Path from an empty string"))
+}
+ }
+}
+
+/**
+ * The class contains tests for the `ALTER NAMESPACE ... SET LOCATION` command
to
+ * check V1 In-Memory table catalog.
+ */
+class AlterNamespaceSetLocationSuite extends AlterNamespaceSetLocationSuiteBase
+with CommandSuiteBase {
+ override def commandVersion: String =
super[AlterNamespaceSetLocationSuiteBase].commandVersion
+
+ test("basic v1 test") {
+val ns = "db1"
+withNamespace(ns) {
+ sql(s"CREATE NAMESPACE $catalog.$ns")
+ withTempDir { tmpDir =>
+sql(s"ALTER NAMESPACE $catalog.$ns SET LOCATION '${tmpDir.toURI}'")
+val sessionCatalog = spark.sessionState.catalog
+val uriInCatalog = sessionCatalog.getDatabaseMetadata(ns).locationUri
Review comment:
yea let's unify it, so that we can move this test to the base trait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34610: [SPARK-34332][SQL][TEST] Unify v1 and v2 ALTER TABLE .. SET LOCATION tests
cloud-fan commented on a change in pull request #34610:
URL: https://github.com/apache/spark/pull/34610#discussion_r749898065
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterNamespaceSetLocationSuite.scala
##
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `ALTER NAMESPACE ... SET
LOCATION` command that
+ * checks V1 table catalogs. The tests that cannot run for all V1 catalogs are
located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog:
+ *
`org.apache.spark.sql.execution.command.v1.AlterNamespaceSetLocationSuite`
+ * - V1 Hive External catalog:
+ *
`org.apache.spark.sql.hive.execution.command.AlterNamespaceSetLocationSuite`
+ */
+trait AlterNamespaceSetLocationSuiteBase extends
command.AlterNamespaceSetLocationSuiteBase
+with command.TestsV1AndV2Commands {
+ override def notFoundMsgPrefix: String = "Database"
+
+ test ("Empty location string") {
Review comment:
```suggestion
test("Empty location string") {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34610: [SPARK-34332][SQL][TEST] Unify v1 and v2 ALTER TABLE .. SET LOCATION tests
cloud-fan commented on a change in pull request #34610:
URL: https://github.com/apache/spark/pull/34610#discussion_r749897957
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterNamespaceSetLocationSuite.scala
##
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `ALTER NAMESPACE ... SET
LOCATION` command that
+ * checks V1 table catalogs. The tests that cannot run for all V1 catalogs are
located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog:
+ *
`org.apache.spark.sql.execution.command.v1.AlterNamespaceSetLocationSuite`
+ * - V1 Hive External catalog:
+ *
`org.apache.spark.sql.hive.execution.command.AlterNamespaceSetLocationSuite`
+ */
+trait AlterNamespaceSetLocationSuiteBase extends
command.AlterNamespaceSetLocationSuiteBase
+with command.TestsV1AndV2Commands {
+ override def notFoundMsgPrefix: String = "Database"
+
+ test ("Empty location string") {
+val ns = "db1"
+withNamespace(ns) {
+ sql(s"CREATE NAMESPACE $catalog.$ns")
+ val message = intercept[IllegalArgumentException] {
+sql(s"ALTER DATABASE $catalog.$ns SET LOCATION ''")
Review comment:
I think most sources will validate the location, maybe we should do it
in the in-memory testing v2 source as well, so that we can move this test to
the base trait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34610: [SPARK-34332][SQL][TEST] Unify v1 and v2 ALTER TABLE .. SET LOCATION tests
cloud-fan commented on a change in pull request #34610:
URL: https://github.com/apache/spark/pull/34610#discussion_r749897588
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
##
@@ -778,18 +778,6 @@ abstract class DDLSuite extends QueryTest with
SQLTestUtils {
Row("Properties", "((a,a), (b,b), (c,c), (d,d))") :: Nil)
withTempDir { tmpDir =>
- if (isUsingHiveMetastore) {
-val e1 = intercept[AnalysisException] {
- sql(s"ALTER DATABASE $dbName SET LOCATION '${tmpDir.toURI}'")
-}
-assert(e1.getMessage.contains("does not support altering database
location"))
- } else {
-sql(s"ALTER DATABASE $dbName SET LOCATION '${tmpDir.toURI}'")
-val uriInCatalog =
catalog.getDatabaseMetadata(dbNameWithoutBackTicks).locationUri
-assert("file" === uriInCatalog.getScheme)
-assert(new Path(tmpDir.getPath).toUri.getPath ===
uriInCatalog.getPath)
- }
-
intercept[NoSuchDatabaseException] {
sql(s"ALTER DATABASE `db-not-exist` SET LOCATION
'${tmpDir.toURI}'")
Review comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
AmplabJenkins removed a comment on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969855677 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145253/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
AmplabJenkins commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969855677 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145253/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
SparkQA removed a comment on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969441918 **[Test build #145253 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145253/testReport)** for PR 34575 at commit [`170378b`](https://github.com/apache/spark/commit/170378bc4747be2d311d420b74e8065b5eef9086). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34575: [SPARK-37273][SQL] Support hidden file metadata columns in Spark SQL
SparkQA commented on pull request #34575: URL: https://github.com/apache/spark/pull/34575#issuecomment-969854776 **[Test build #145253 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145253/testReport)** for PR 34575 at commit [`170378b`](https://github.com/apache/spark/commit/170378bc4747be2d311d420b74e8065b5eef9086). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34607: [SPARK-36038][CORE] Speculation metrics summary at stage level
AmplabJenkins removed a comment on pull request #34607: URL: https://github.com/apache/spark/pull/34607#issuecomment-969852748 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145260/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34607: [SPARK-36038][CORE] Speculation metrics summary at stage level
AmplabJenkins commented on pull request #34607: URL: https://github.com/apache/spark/pull/34607#issuecomment-969852748 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145260/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dchvn edited a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
dchvn edited a comment on pull request #34060: URL: https://github.com/apache/spark/pull/34060#issuecomment-969851943 Hi @huaxingao , any update? I am trying to migrate `ReplaceTableStatement` and `CreateTableAsSelectStatement` by learn from your PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34607: [SPARK-36038][CORE] Speculation metrics summary at stage level
SparkQA removed a comment on pull request #34607: URL: https://github.com/apache/spark/pull/34607#issuecomment-969650454 **[Test build #145260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145260/testReport)** for PR 34607 at commit [`ea6c7be`](https://github.com/apache/spark/commit/ea6c7be75c1b08407686680b112025f6e995e87b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
dchvn commented on pull request #34060: URL: https://github.com/apache/spark/pull/34060#issuecomment-969851943 Hi @huaxingao , any update? I am trying to migrate `ReplaceTableStatement` by learn from your PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34607: [SPARK-36038][CORE] Speculation metrics summary at stage level
SparkQA commented on pull request #34607: URL: https://github.com/apache/spark/pull/34607#issuecomment-969851814 **[Test build #145260 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145260/testReport)** for PR 34607 at commit [`ea6c7be`](https://github.com/apache/spark/commit/ea6c7be75c1b08407686680b112025f6e995e87b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #34072: [SPARK-36680][SQL] Supports Dynamic Table Options for Spark SQL
HyukjinKwon edited a comment on pull request #34072: URL: https://github.com/apache/spark/pull/34072#issuecomment-969849020 Can we fix the PR description better for that point? There seems already a way to resolve the issue PR description explains. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34497: [SPARK-37219][SQL] Add AS OF syntax support
SparkQA commented on pull request #34497: URL: https://github.com/apache/spark/pull/34497#issuecomment-969849023 **[Test build #145266 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145266/testReport)** for PR 34497 at commit [`c852124`](https://github.com/apache/spark/commit/c852124d5d510472527c4babe8008a9927a587ba). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
