[GitHub] [spark] SparkQA commented on issue #27644: [SPARK-30868][SQL] Throw Exception if runHive(sql) failed
SparkQA commented on issue #27644: [SPARK-30868][SQL] Throw Exception if runHive(sql) failed URL: https://github.com/apache/spark/pull/27644#issuecomment-589541503 **[Test build #118756 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118756/testReport)** for PR 27644 at commit [`6d7435e`](https://github.com/apache/spark/commit/6d7435ed11e502c8ae517f33eaf6fb2ea12aae29). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27644: [SPARK-30868][SQL] Throw Exception if runHive(sql) failed
SparkQA removed a comment on issue #27644: [SPARK-30868][SQL] Throw Exception if runHive(sql) failed URL: https://github.com/apache/spark/pull/27644#issuecomment-589510776 **[Test build #118756 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118756/testReport)** for PR 27644 at commit [`6d7435e`](https://github.com/apache/spark/commit/6d7435ed11e502c8ae517f33eaf6fb2ea12aae29). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27644: [SPARK-30868][SQL] Throw Exception if runHive(sql) failed
AmplabJenkins commented on issue #27644: [SPARK-30868][SQL] Throw Exception if runHive(sql) failed URL: https://github.com/apache/spark/pull/27644#issuecomment-589541605 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27644: [SPARK-30868][SQL] Throw Exception if runHive(sql) failed
AmplabJenkins removed a comment on issue #27644: [SPARK-30868][SQL] Throw Exception if runHive(sql) failed URL: https://github.com/apache/spark/pull/27644#issuecomment-589541605 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27644: [SPARK-30868][SQL] Throw Exception if runHive(sql) failed
AmplabJenkins commented on issue #27644: [SPARK-30868][SQL] Throw Exception if runHive(sql) failed URL: https://github.com/apache/spark/pull/27644#issuecomment-589541612 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118756/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments
cloud-fan commented on issue #27495: [SPARK-28880][SQL] Support ANSI nested
bracketed comments
URL: https://github.com/apache/spark/pull/27495#issuecomment-589540986
I have a new idea. Without considering hint, the rule to match comment is
very simple:
```
'/*' (BRACKETED_COMMENT|.)*? '*/' -> channel(HIDDEN)
```
And there is a better way to check if it's hint
```
'/*' (BRACKETED_COMMENT|.)*? '*/' {isHint()}? -> channel(HIDDEN)
```
We can follow `isValidDecimal()` in `Sqlbase.g4` to implement `isHint()`,
which checks the first 3 chars and see if they are "/*-"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs
AmplabJenkins commented on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs URL: https://github.com/apache/spark/pull/27560#issuecomment-589539592 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs
AmplabJenkins removed a comment on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs URL: https://github.com/apache/spark/pull/27560#issuecomment-589539598 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118749/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs
AmplabJenkins commented on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs URL: https://github.com/apache/spark/pull/27560#issuecomment-589539598 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118749/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs
AmplabJenkins removed a comment on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs URL: https://github.com/apache/spark/pull/27560#issuecomment-589539592 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs
SparkQA removed a comment on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs URL: https://github.com/apache/spark/pull/27560#issuecomment-589493045 **[Test build #118749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118749/testReport)** for PR 27560 at commit [`14b121e`](https://github.com/apache/spark/commit/14b121e28ee9695ff1d8ac1c2731f3250e6590a8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs
SparkQA commented on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs URL: https://github.com/apache/spark/pull/27560#issuecomment-589538924 **[Test build #118749 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118749/testReport)** for PR 27560 at commit [`14b121e`](https://github.com/apache/spark/commit/14b121e28ee9695ff1d8ac1c2731f3250e6590a8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27655: [SPARK-30897][SQL] The behavior of ArrayExists should not depend on SQLConf.get
AmplabJenkins commented on issue #27655: [SPARK-30897][SQL] The behavior of ArrayExists should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27655#issuecomment-589538601 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23514/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27655: [SPARK-30897][SQL] The behavior of ArrayExists should not depend on SQLConf.get
AmplabJenkins removed a comment on issue #27655: [SPARK-30897][SQL] The behavior of ArrayExists should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27655#issuecomment-589538601 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23514/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27655: [SPARK-30897][SQL] The behavior of ArrayExists should not depend on SQLConf.get
AmplabJenkins removed a comment on issue #27655: [SPARK-30897][SQL] The behavior of ArrayExists should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27655#issuecomment-589538595 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27655: [SPARK-30897][SQL] The behavior of ArrayExists should not depend on SQLConf.get
AmplabJenkins commented on issue #27655: [SPARK-30897][SQL] The behavior of ArrayExists should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27655#issuecomment-589538595 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27655: [SPARK-30897][SQL] The behavior of ArrayExists should not depend on SQLConf.get
SparkQA commented on issue #27655: [SPARK-30897][SQL] The behavior of ArrayExists should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27655#issuecomment-589538273 **[Test build #118763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118763/testReport)** for PR 27655 at commit [`87aac55`](https://github.com/apache/spark/commit/87aac55538108f2007aabc0286a463104231f74e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #27661: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision
HyukjinKwon closed pull request #27661: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision URL: https://github.com/apache/spark/pull/27661 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #27655: [SPARK-30897][SQL] The behavior of ArrayExists should not depend on SQLConf.get
peter-toth commented on a change in pull request #27655: [SPARK-30897][SQL] The
behavior of ArrayExists should not depend on SQLConf.get
URL: https://github.com/apache/spark/pull/27655#discussion_r382435704
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala
##
@@ -522,11 +522,14 @@ case class ArrayFilter(
since = "2.4.0")
case class ArrayExists(
argument: Expression,
-function: Expression)
+function: Expression,
+followThreeValuedLogic: Boolean)
extends ArrayBasedSimpleHigherOrderFunction with CodegenFallback {
- private val followThreeValuedLogic =
-SQLConf.get.getConf(SQLConf.LEGACY_ARRAY_EXISTS_FOLLOWS_THREE_VALUED_LOGIC)
+ def this(argument: Expression, function: Expression) = {
+this(argument, function,
+
SQLConf.get.getConf(SQLConf.LEGACY_ARRAY_EXISTS_FOLLOWS_THREE_VALUED_LOGIC))
Review comment:
Thanks, fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
cloud-fan commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382433518
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##
@@ -104,7 +105,17 @@ case class DataSourceAnalysis(conf: SQLConf) extends
Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
-Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+conf.storeAssignmentPolicy match {
+ // SPARK-30844: try our best to follow StoreAssignmentPolicy for
static partition
+ // values but not completely follow because we can't
use`DataType.canWrite` due to
Review comment:
nit: be more general like `... because we can't do static type checking due
to ...`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table URL: https://github.com/apache/spark/pull/27597#issuecomment-589534209 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23513/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table URL: https://github.com/apache/spark/pull/27597#issuecomment-589534205 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table URL: https://github.com/apache/spark/pull/27597#issuecomment-589534205 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table URL: https://github.com/apache/spark/pull/27597#issuecomment-589534209 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23513/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27658: [SPARK-30894][SQL] Make Size's nullable independent from SQL config changes
cloud-fan closed pull request #27658: [SPARK-30894][SQL] Make Size's nullable independent from SQL config changes URL: https://github.com/apache/spark/pull/27658 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27658: [SPARK-30894][SQL] Make Size's nullable independent from SQL config changes
cloud-fan commented on issue #27658: [SPARK-30894][SQL] Make Size's nullable independent from SQL config changes URL: https://github.com/apache/spark/pull/27658#issuecomment-589534103 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table URL: https://github.com/apache/spark/pull/27597#issuecomment-589533846 **[Test build #118762 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118762/testReport)** for PR 27597 at commit [`a70c89c`](https://github.com/apache/spark/commit/a70c89c1138bd1e643fa95de8cad7c0c152ff76b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
Ngone51 commented on a change in pull request #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382432246
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##
@@ -104,7 +105,13 @@ case class DataSourceAnalysis(conf: SQLConf) extends
Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
-Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+conf.storeAssignmentPolicy match {
+ case StoreAssignmentPolicy.ANSI | StoreAssignmentPolicy.STRICT =>
Review comment:
Added, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow
cloud-fan commented on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow URL: https://github.com/apache/spark/pull/27627#issuecomment-589530921 > Sum does not take care of possibility of overflow for the intermediate steps. ie the updateExpressions and mergeExpressions. I'm a little confused. These expressions are used in non-whole-stage-codegen as well, why only whole-stage-codegen has the problem? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27662: [MINOR][SQL] Fix error position of NOSCAN
AmplabJenkins commented on issue #27662: [MINOR][SQL] Fix error position of NOSCAN URL: https://github.com/apache/spark/pull/27662#issuecomment-589529896 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27662: [MINOR][SQL] Fix error position of NOSCAN
AmplabJenkins removed a comment on issue #27662: [MINOR][SQL] Fix error position of NOSCAN URL: https://github.com/apache/spark/pull/27662#issuecomment-589529900 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118743/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27662: [MINOR][SQL] Fix error position of NOSCAN
AmplabJenkins commented on issue #27662: [MINOR][SQL] Fix error position of NOSCAN URL: https://github.com/apache/spark/pull/27662#issuecomment-589529900 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118743/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27662: [MINOR][SQL] Fix error position of NOSCAN
AmplabJenkins removed a comment on issue #27662: [MINOR][SQL] Fix error position of NOSCAN URL: https://github.com/apache/spark/pull/27662#issuecomment-589529896 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27662: [MINOR][SQL] Fix error position of NOSCAN
SparkQA removed a comment on issue #27662: [MINOR][SQL] Fix error position of NOSCAN URL: https://github.com/apache/spark/pull/27662#issuecomment-589468236 **[Test build #118743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118743/testReport)** for PR 27662 at commit [`d467c7d`](https://github.com/apache/spark/commit/d467c7d27a448742b8738c99d59d8f337b63852a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27662: [MINOR][SQL] Fix error position of NOSCAN
SparkQA commented on issue #27662: [MINOR][SQL] Fix error position of NOSCAN URL: https://github.com/apache/spark/pull/27662#issuecomment-589529295 **[Test build #118743 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118743/testReport)** for PR 27662 at commit [`d467c7d`](https://github.com/apache/spark/commit/d467c7d27a448742b8738c99d59d8f337b63852a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
AmplabJenkins removed a comment on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27657#issuecomment-589526227 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118761/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
AmplabJenkins commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27657#issuecomment-589526227 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118761/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
AmplabJenkins removed a comment on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27657#issuecomment-589526219 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
AmplabJenkins commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27657#issuecomment-589526219 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
SparkQA removed a comment on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27657#issuecomment-589521877 **[Test build #118761 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118761/testReport)** for PR 27657 at commit [`7bbf40e`](https://github.com/apache/spark/commit/7bbf40e1451f96eb97a479d407735c81f6742279). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
SparkQA commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27657#issuecomment-589526122 **[Test build #118761 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118761/testReport)** for PR 27657 at commit [`7bbf40e`](https://github.com/apache/spark/commit/7bbf40e1451f96eb97a479d407735c81f6742279). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class CreateArray(children: Seq[Expression], useStringTypeWhenEmpty: Boolean)` * `case class CreateMap(children: Seq[Expression], useStringTypeWhenEmpty: Boolean)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
Ngone51 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default URL: https://github.com/apache/spark/pull/27488#issuecomment-589525950 thanks all! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27662: [MINOR][SQL] Fix error position of NOSCAN
Ngone51 commented on issue #27662: [MINOR][SQL] Fix error position of NOSCAN URL: https://github.com/apache/spark/pull/27662#issuecomment-589526023 thanks all! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue"
Ngone51 commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue" URL: https://github.com/apache/spark/pull/27540#issuecomment-589525865 thanks all!! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
cloud-fan commented on a change in pull request #27657: [SPARK-30899]:
CreateArray/CreateMap's data type should not depend on SQLConf.get
URL: https://github.com/apache/spark/pull/27657#discussion_r382423049
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
##
@@ -606,7 +610,9 @@ object TypeCoercion {
case None => m.values
}
-CreateMap(newKeys.zip(newValues).flatMap { case (k, v) => Seq(k, v) })
+CreateMap(
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
cloud-fan commented on a change in pull request #27657: [SPARK-30899]:
CreateArray/CreateMap's data type should not depend on SQLConf.get
URL: https://github.com/apache/spark/pull/27657#discussion_r382422992
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
##
@@ -553,10 +553,14 @@ object TypeCoercion {
// Skip nodes who's children have not been resolved yet.
case e if !e.childrenResolved => e
- case a @ CreateArray(children) if
!haveSameType(children.map(_.dataType)) =>
+ case a @ CreateArray(children, useStringTypeWhenEmpty)
+if !haveSameType(children.map(_.dataType)) =>
val types = children.map(_.dataType)
findWiderCommonType(types) match {
- case Some(finalDataType) =>
CreateArray(children.map(castIfNotSameType(_, finalDataType)))
+ case Some(finalDataType) => CreateArray(
Review comment:
nit: `a.copy(children = ...)`, then we don't need the
`useStringTypeWhenEmpty` parameter and can just write `case a @
CreateArray(children, _)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27661: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision
HyukjinKwon commented on issue #27661: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision URL: https://github.com/apache/spark/pull/27661#issuecomment-589523563 Let's close and review #27618 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
cloud-fan commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default URL: https://github.com/apache/spark/pull/27488#issuecomment-589522387 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
cloud-fan closed pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default URL: https://github.com/apache/spark/pull/27488 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
SparkQA commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27657#issuecomment-589521877 **[Test build #118761 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118761/testReport)** for PR 27657 at commit [`7bbf40e`](https://github.com/apache/spark/commit/7bbf40e1451f96eb97a479d407735c81f6742279). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27604: [SPARK-30849][CORE][SHUFFLE]Fix application failed due to failed to get MapStatuses broadcast block
Ngone51 commented on a change in pull request #27604:
[SPARK-30849][CORE][SHUFFLE]Fix application failed due to failed to get
MapStatuses broadcast block
URL: https://github.com/apache/spark/pull/27604#discussion_r382411144
##
File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala
##
@@ -824,11 +825,15 @@ private[spark] class MapOutputTrackerWorker(conf:
SparkConf) extends MapOutputTr
endPartition: Int): Iterator[(BlockManagerId, Seq[(BlockId, Long,
Int)])] = {
logDebug(s"Fetching outputs for shuffle $shuffleId, mappers
$startMapIndex-$endMapIndex" +
s"partitions $startPartition-$endPartition")
-val statuses = getStatuses(shuffleId, conf)
try {
+ val statuses = getStatuses(shuffleId, conf)
MapOutputTracker.convertMapStatuses(
shuffleId, startPartition, endPartition, statuses, startMapIndex,
endMapIndex)
} catch {
+ case e: IOException if
+
Throwables.getCausalChain(e).asScala.exists(_.isInstanceOf[BlockNotFoundException])
=>
+mapStatuses.clear()
+throw new MetadataFetchFailedException(shuffleId, -1,
Throwables.getStackTraceAsString(e))
Review comment:
I'm wondering that we may could add a new type of `FetchFailedException` to
indicate the whole `MapStatus` lost rather than a piece of it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27604: [SPARK-30849][CORE][SHUFFLE]Fix application failed due to failed to get MapStatuses broadcast block
Ngone51 commented on a change in pull request #27604:
[SPARK-30849][CORE][SHUFFLE]Fix application failed due to failed to get
MapStatuses broadcast block
URL: https://github.com/apache/spark/pull/27604#discussion_r382412090
##
File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala
##
@@ -824,11 +825,15 @@ private[spark] class MapOutputTrackerWorker(conf:
SparkConf) extends MapOutputTr
endPartition: Int): Iterator[(BlockManagerId, Seq[(BlockId, Long,
Int)])] = {
logDebug(s"Fetching outputs for shuffle $shuffleId, mappers
$startMapIndex-$endMapIndex" +
s"partitions $startPartition-$endPartition")
-val statuses = getStatuses(shuffleId, conf)
try {
+ val statuses = getStatuses(shuffleId, conf)
Review comment:
I'd prefer to add another try catch when we call `broadcast.vaule` at:
https://github.com/apache/spark/blob/a2aa966ef64bc06f65a646777568427d360605e9/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L969
and throw `FetchFailedException` when we catch `SparkException` from
broadcast.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
AmplabJenkins commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27657#issuecomment-589520239 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23512/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
AmplabJenkins commented on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27657#issuecomment-589520236 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27604: [SPARK-30849][CORE][SHUFFLE]Fix application failed due to failed to get MapStatuses broadcast block
Ngone51 commented on a change in pull request #27604: [SPARK-30849][CORE][SHUFFLE]Fix application failed due to failed to get MapStatuses broadcast block URL: https://github.com/apache/spark/pull/27604#discussion_r382410631 ## File path: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ## @@ -186,7 +186,7 @@ private[spark] class TorrentBroadcast[T: ClassTag](obj: T, id: Long) } blocks(pid) = new ByteBufferBlockData(b, true) case None => - throw new SparkException(s"Failed to get $pieceId of $broadcastId") Review comment: Why not just catch `SparkException` outside? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
AmplabJenkins removed a comment on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27657#issuecomment-589520236 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get
AmplabJenkins removed a comment on issue #27657: [SPARK-30899]: CreateArray/CreateMap's data type should not depend on SQLConf.get URL: https://github.com/apache/spark/pull/27657#issuecomment-589520239 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23512/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
SparkQA commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589519875 **[Test build #118760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118760/testReport)** for PR 27659 at commit [`95ddfab`](https://github.com/apache/spark/commit/95ddfab6e953692cf4f1fcc6d2c523209c6ba2a3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
AmplabJenkins removed a comment on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589518224 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
AmplabJenkins commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589518226 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23511/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
AmplabJenkins commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589518224 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
AmplabJenkins removed a comment on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589518226 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23511/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
cloud-fan commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589517992 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision
cloud-fan commented on a change in pull request #27618: [SPARK-30869][SQL]
Convert dates to/from timestamps in microseconds precision
URL: https://github.com/apache/spark/pull/27618#discussion_r382416037
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
##
@@ -608,16 +608,16 @@ class DateTimeUtilsSuite extends SparkFunSuite with
Matchers with SQLHelper {
val skipped = skipped_days.getOrElse(tz.getID, Set.empty)
(-2 to 2).foreach { d =>
if (!skipped.contains(d)) {
- assert(millisToDays(daysToMillis(d, tz.toZoneId), tz.toZoneId) === d,
+ assert(microsToDays(daysToMicros(d, tz.toZoneId), tz.toZoneId) === d,
s"Round trip of ${d} did not work in tz ${tz}")
}
}
}
}
test("toMillis") {
Review comment:
`microsToMillis`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
AmplabJenkins removed a comment on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589516919 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118753/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
AmplabJenkins removed a comment on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589516913 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision
cloud-fan commented on a change in pull request #27618: [SPARK-30869][SQL]
Convert dates to/from timestamps in microseconds precision
URL: https://github.com/apache/spark/pull/27618#discussion_r382415458
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -88,14 +86,14 @@ object DateTimeUtils {
* Returns the number of days since epoch from java.sql.Date.
*/
def fromJavaDate(date: Date): SQLDate = {
-millisToDays(date.getTime)
+microsToDays(Math.multiplyExact(date.getTime, MICROS_PER_MILLIS))
}
/**
* Returns a java.sql.Date from number of days since epoch.
*/
def toJavaDate(daysSinceEpoch: SQLDate): Date = {
-new Date(daysToMillis(daysSinceEpoch))
+new Date(daysToMicros(daysSinceEpoch) / MICROS_PER_MILLIS)
Review comment:
ditto, `microsToMillis(daysToMicros(daysSinceEpoch))`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
SparkQA commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589516807 **[Test build #118753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118753/testReport)** for PR 27659 at commit [`95ddfab`](https://github.com/apache/spark/commit/95ddfab6e953692cf4f1fcc6d2c523209c6ba2a3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
AmplabJenkins commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589516919 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118753/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision
cloud-fan commented on a change in pull request #27618: [SPARK-30869][SQL]
Convert dates to/from timestamps in microseconds precision
URL: https://github.com/apache/spark/pull/27618#discussion_r382415344
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -88,14 +86,14 @@ object DateTimeUtils {
* Returns the number of days since epoch from java.sql.Date.
*/
def fromJavaDate(date: Date): SQLDate = {
-millisToDays(date.getTime)
+microsToDays(Math.multiplyExact(date.getTime, MICROS_PER_MILLIS))
Review comment:
shall we call `millisToMicros(date.getTime)` instead of
`Math.multiplyExact(date.getTime, MICROS_PER_MILLIS)`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
SparkQA removed a comment on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589499680 **[Test build #118753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118753/testReport)** for PR 27659 at commit [`95ddfab`](https://github.com/apache/spark/commit/95ddfab6e953692cf4f1fcc6d2c523209c6ba2a3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe
AmplabJenkins commented on issue #27659: [SPARK-30906][SQL] Turning off AQE in CacheManager is not thread-safe URL: https://github.com/apache/spark/pull/27659#issuecomment-589516913 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #27662: [MINOR][SQL] Fix error position of NOSCAN
maropu commented on issue #27662: [MINOR][SQL] Fix error position of NOSCAN URL: https://github.com/apache/spark/pull/27662#issuecomment-589516586 Thanks! Merged to master/3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-589516320 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-589516325 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23510/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision
AmplabJenkins commented on issue #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision URL: https://github.com/apache/spark/pull/27618#issuecomment-589516297 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23509/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision
AmplabJenkins commented on issue #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision URL: https://github.com/apache/spark/pull/27618#issuecomment-589516294 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-589516325 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23510/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision
AmplabJenkins removed a comment on issue #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision URL: https://github.com/apache/spark/pull/27618#issuecomment-589516294 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu closed pull request #27662: [MINOR][SQL] Fix error position of NOSCAN
maropu closed pull request #27662: [MINOR][SQL] Fix error position of NOSCAN URL: https://github.com/apache/spark/pull/27662 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision
AmplabJenkins removed a comment on issue #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision URL: https://github.com/apache/spark/pull/27618#issuecomment-589516297 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23509/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-589516320 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL]
Refactor DateTimeUtils
URL: https://github.com/apache/spark/pull/27617#discussion_r382414618
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
##
@@ -144,7 +144,7 @@ class LegacyFastTimestampFormatter(
cal.getTimeInMillis * MICROS_PER_MILLIS + micros
}
- def format(timestamp: SQLTimestamp): String = {
+ def format(timestamp: Long): String = {
Review comment:
`microsec`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision
SparkQA commented on issue #27618: [SPARK-30869][SQL] Convert dates to/from timestamps in microseconds precision URL: https://github.com/apache/spark/pull/27618#issuecomment-589516028 **[Test build #118758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118758/testReport)** for PR 27618 at commit [`f78d773`](https://github.com/apache/spark/commit/f78d773c41b7843c1979f7a462384b9339f30b51). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
SparkQA commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-589516013 **[Test build #118759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118759/testReport)** for PR 27617 at commit [`c475f27`](https://github.com/apache/spark/commit/c475f271b24f8d9ef7bbbc0f93a238600d70a717). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL]
Refactor DateTimeUtils
URL: https://github.com/apache/spark/pull/27617#discussion_r382414377
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -407,53 +398,53 @@ object DateTimeUtils {
}
}
- private def localTimestamp(microsec: SQLTimestamp, zoneId: ZoneId):
LocalDateTime = {
+ private def localTimestamp(microsec: Long, zoneId: ZoneId): LocalDateTime = {
microsToInstant(microsec).atZone(zoneId).toLocalDateTime
}
/**
* Returns the hour value of a given timestamp value. The timestamp is
expressed in microseconds.
*/
- def getHours(microsec: SQLTimestamp, zoneId: ZoneId): Int = {
+ def getHours(microsec: Long, zoneId: ZoneId): Int = {
localTimestamp(microsec, zoneId).getHour
}
/**
* Returns the minute value of a given timestamp value. The timestamp is
expressed in
* microseconds.
*/
- def getMinutes(microsec: SQLTimestamp, zoneId: ZoneId): Int = {
+ def getMinutes(microsec: Long, zoneId: ZoneId): Int = {
localTimestamp(microsec, zoneId).getMinute
}
/**
* Returns the second value of a given timestamp value. The timestamp is
expressed in
* microseconds.
*/
- def getSeconds(microsec: SQLTimestamp, zoneId: ZoneId): Int = {
+ def getSeconds(microsec: Long, zoneId: ZoneId): Int = {
localTimestamp(microsec, zoneId).getSecond
}
/**
* Returns the seconds part and its fractional part with microseconds.
*/
- def getSecondsWithFraction(microsec: SQLTimestamp, zoneId: ZoneId): Decimal
= {
+ def getSecondsWithFraction(microsec: Long, zoneId: ZoneId): Decimal = {
Decimal(getMicroseconds(microsec, zoneId), 8, 6)
}
/**
* Returns seconds, including fractional parts, multiplied by 1000. The
timestamp
* is expressed in microseconds since the epoch.
*/
- def getMilliseconds(timestamp: SQLTimestamp, zoneId: ZoneId): Decimal = {
+ def getMilliseconds(timestamp: Long, zoneId: ZoneId): Decimal = {
Review comment:
shall we be consistent and use parameter name "microsec"?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27651: [SPARK-30903][SQL] Fail fast on duplicate columns when analyze columns
maropu commented on a change in pull request #27651: [SPARK-30903][SQL] Fail
fast on duplicate columns when analyze columns
URL: https://github.com/apache/spark/pull/27651#discussion_r382414058
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##
@@ -3202,8 +3202,16 @@ class AstBuilder(conf: SQLConf) extends
SqlBaseBaseVisitor[AnyRef] with Logging
AnalyzeTableStatement(tableName, partitionSpec, noScan = ctx.identifier
!= null)
} else {
checkPartitionSpec()
- AnalyzeColumnStatement(
-tableName, Option(visitIdentifierSeq(ctx.identifierSeq())), allColumns
= false)
+ val colsOpt = Option(visitIdentifierSeq(ctx.identifierSeq()))
+ colsOpt.foreach { cols =>
+val duplicates = cols.groupBy(identity).filter(_._2.size > 1).keys
+if (duplicates.nonEmpty) {
+ throw new ParseException(
Review comment:
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27651: [SPARK-30903][SQL] Fail fast on duplicate columns when analyze columns
maropu commented on a change in pull request #27651: [SPARK-30903][SQL] Fail
fast on duplicate columns when analyze columns
URL: https://github.com/apache/spark/pull/27651#discussion_r382414020
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##
@@ -650,4 +651,17 @@ class StatisticsCollectionSuite extends
StatisticsCollectionTestBase with Shared
}
}
}
+
+ test("SPARK-30903: Fail fast on duplicate columns when analyze columns") {
+val table = "test_table"
+withTable(table) {
+ sql(s"""
+ |CREATE TABLE $table (value string, name string)
+ |USING PARQUET""".stripMargin)
+ val errorMsg = intercept[ParseException] {
+sql(s"ANALYZE TABLE $table COMPUTE STATISTICS FOR COLUMNS value, name,
name, value")
Review comment:
Yea, we need some refactoring for this use... the refactoring itself looks
ok to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL]
Refactor DateTimeUtils
URL: https://github.com/apache/spark/pull/27617#discussion_r382413997
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -138,25 +129,25 @@ object DateTimeUtils {
* Converts the timestamp to milliseconds since epoch. In spark timestamp
values have microseconds
* precision, so this conversion is lossy.
*/
- def toMillis(us: SQLTimestamp): Long = {
+ def toMillis(us: Long): Long = {
// When the timestamp is negative i.e before 1970, we need to adjust the
millseconds portion.
// Example - 1965-01-01 10:11:12.123456 is represented as
(-157700927876544) in micro precision.
// In millis precision the above needs to be represented as
(-157700927877).
Math.floorDiv(us, MICROS_PER_MILLIS)
}
/*
- * Converts milliseconds since epoch to SQLTimestamp.
+ * Converts milliseconds since the epoch to microseconds.
*/
- def fromMillis(millis: Long): SQLTimestamp = {
+ def fromMillis(millis: Long): Long = {
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] iRakson commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions
iRakson commented on issue #27580: [SPARK-27619][SQL]MapType should be prohibited in hash expressions URL: https://github.com/apache/spark/pull/27580#issuecomment-589515559 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
cloud-fan commented on a change in pull request #27617: [SPARK-30865][SQL]
Refactor DateTimeUtils
URL: https://github.com/apache/spark/pull/27617#discussion_r382413974
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -138,25 +129,25 @@ object DateTimeUtils {
* Converts the timestamp to milliseconds since epoch. In spark timestamp
values have microseconds
* precision, so this conversion is lossy.
*/
- def toMillis(us: SQLTimestamp): Long = {
+ def toMillis(us: Long): Long = {
Review comment:
nit: shall we make the name clearer? `microsToMillis`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] iRakson commented on issue #27648: [SPARK-24994][SQL] : Support filter pushdown for short and byte without explicit casting
iRakson commented on issue #27648: [SPARK-24994][SQL] : Support filter pushdown for short and byte without explicit casting URL: https://github.com/apache/spark/pull/27648#issuecomment-589515318 > Is that pushdown legal? if `id` is short, `id = 2` is parsed as `cast(id as int) = 2`. So, I feel that the semantics are different between them. Naive users might not cast `literal` to `shortType` when passing a value within short range. Also, if there are large number of queries which deals with short and byte fields, it is cumbersome to always explicitly cast them. In those cases users can't get push-down benefits. In my opinion we should push the filter if `literals` are in `short`/`byte` range. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-589512791 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-589512797 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23508/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins removed a comment on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-589512791 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27651: [SPARK-30903][SQL] Fail fast on duplicate columns when analyze columns
cloud-fan commented on a change in pull request #27651: [SPARK-30903][SQL] Fail
fast on duplicate columns when analyze columns
URL: https://github.com/apache/spark/pull/27651#discussion_r382411322
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##
@@ -3202,8 +3202,16 @@ class AstBuilder(conf: SQLConf) extends
SqlBaseBaseVisitor[AnyRef] with Logging
AnalyzeTableStatement(tableName, partitionSpec, noScan = ctx.identifier
!= null)
} else {
checkPartitionSpec()
- AnalyzeColumnStatement(
-tableName, Option(visitIdentifierSeq(ctx.identifierSeq())), allColumns
= false)
+ val colsOpt = Option(visitIdentifierSeq(ctx.identifierSeq()))
+ colsOpt.foreach { cols =>
+val duplicates = cols.groupBy(identity).filter(_._2.size > 1).keys
+if (duplicates.nonEmpty) {
+ throw new ParseException(
Review comment:
shall we check it in `CheckAnalysis`? It's not a parser error really
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-589512797 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23508/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
SparkQA commented on issue #27617: [SPARK-30865][SQL] Refactor DateTimeUtils URL: https://github.com/apache/spark/pull/27617#issuecomment-589512485 **[Test build #118757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118757/testReport)** for PR 27617 at commit [`9ba2d3d`](https://github.com/apache/spark/commit/9ba2d3d4ceb8f7b863ef0887f4e2c879a4ef05d4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
cloud-fan commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382410965
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##
@@ -104,7 +105,13 @@ case class DataSourceAnalysis(conf: SQLConf) extends
Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
-Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+conf.storeAssignmentPolicy match {
+ case StoreAssignmentPolicy.ANSI | StoreAssignmentPolicy.STRICT =>
Review comment:
yea let's add some comments
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #27648: [SPARK-24994][SQL] : Support filter pushdown for short and byte without explicit casting
maropu commented on issue #27648: [SPARK-24994][SQL] : Support filter pushdown for short and byte without explicit casting URL: https://github.com/apache/spark/pull/27648#issuecomment-589512300 > 2 is within short range still filter is not pushed down. > But it pushes filter if we cast the literal to smallint: > select * from t where id = cast(2 as smallint) ; Is that pushdown legal? if `id` is short, `id = 2` is parsed as `cast(id as int) = 2`. So, I feel that the semantics are different between them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
