[GitHub] [spark] SparkQA removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
SparkQA removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501105447 **[Test build #106401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106401/testReport)** for PR 24792 at commit [`a912b87`](https://github.com/apache/spark/commit/a912b87893d22e603d334429b3ca3de644ca2780). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
AmplabJenkins commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501139378 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501140964 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
Ngone51 commented on a change in pull request #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#discussion_r292763964 ## File path: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala ## @@ -220,6 +225,38 @@ private[deploy] class Worker( metricsSystem.getServletHandlers.foreach(webUi.attachHandler) } + // TODO if we're starting up multi workers under the same host, discovery script won't work. Review comment: For the initial version, I have a base assumption here that user should be responsible for configuring the right `resourceFile` among multi workers on the same hosts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501140964 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501140976 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11649/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501140976 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11649/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
cloud-fan commented on a change in pull request #24806: [WIP][SPARK-27856][SQL]
Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292767749
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DataSourceV2AnalysisSuite.scala
##
@@ -440,16 +439,15 @@ abstract class DataSourceV2AnalysisSuite extends
AnalysisTest {
StructField("y", DoubleType))).toAttributes)
val query = TestRelation(StructType(Seq(
- StructField("x", DoubleType),
+ StructField("x", FloatType),
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
cloud-fan commented on a change in pull request #24806: [WIP][SPARK-27856][SQL]
Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292767723
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DataSourceV2AnalysisSuite.scala
##
@@ -290,15 +290,14 @@ abstract class DataSourceV2AnalysisSuite extends
AnalysisTest {
StructField("y", DoubleType))).toAttributes)
val query = TestRelation(StructType(Seq(
- StructField("x", DoubleType),
+ StructField("x", FloatType),
Review comment:
why this change?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
cloud-fan commented on a change in pull request #24806: [WIP][SPARK-27856][SQL]
Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292768054
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/EncoderResolutionSuite.scala
##
@@ -230,11 +230,12 @@ class EncoderResolutionSuite extends PlanTest {
castSuccess[Long, String]
castSuccess[Int, java.math.BigDecimal]
castSuccess[Long, java.math.BigDecimal]
+ castSuccess[Double, java.math.BigDecimal]
Review comment:
How is this supported?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501148069 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106402/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501147903 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106404/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases
AmplabJenkins removed a comment on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases URL: https://github.com/apache/spark/pull/24842#issuecomment-501147926 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106405/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed.
AmplabJenkins removed a comment on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed. URL: https://github.com/apache/spark/pull/24848#issuecomment-501163279 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed.
AmplabJenkins removed a comment on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed. URL: https://github.com/apache/spark/pull/24848#issuecomment-501164921 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed.
AmplabJenkins commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed. URL: https://github.com/apache/spark/pull/24848#issuecomment-501164921 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed.
AmplabJenkins commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed. URL: https://github.com/apache/spark/pull/24848#issuecomment-501165141 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float
AmplabJenkins commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float URL: https://github.com/apache/spark/pull/24849#issuecomment-501167829 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11656/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float
AmplabJenkins commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float URL: https://github.com/apache/spark/pull/24849#issuecomment-501167823 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float
gengliangwang commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float URL: https://github.com/apache/spark/pull/24849#issuecomment-501168100 @cloud-fan @ueshin @maropu @dongjoon-hyun @rxin @rdblue @gatorsmile Actually, I am not super confident about this. I would like to know your idea. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum opened a new pull request #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
wangyum opened a new pull request #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql URL: https://github.com/apache/spark/pull/24850 ## What changes were proposed in this pull request? This PR is to port date.sql from PostgreSQL regression tests. https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/date.sql The expected results can be found in the link: https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/expected/date.out When porting the test cases, found two PostgreSQL specific features that do not exist in Spark SQL: [SPARK-27898](https://issues.apache.org/jira/browse/SPARK-27898): Support 4 date operators(date + integer, integer + date, date - integer and date - date) [SPARK-28017](https://issues.apache.org/jira/browse/SPARK-28017): Enhance EXTRACT/DATE_TRUNC Also, found a bug: [SPARK-28015](https://issues.apache.org/jira/browse/SPARK-28015): Invalid date formats should throw an exception Also, found a inconsistent behavior: [SPARK-27923](https://issues.apache.org/jira/browse/SPARK-27923): Invalid date throw an exception bug Spark SQL returns `NULL`, for example: https://github.com/postgres/postgres/blob/30bcebbdcf23eb8b78e553c4b3b5eb847410ef19/src/test/regress/expected/date.out#L13-L14 ## How was this patch tested? N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
SparkQA commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501183254 **[Test build #106410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106410/testReport)** for PR 24735 at commit [`053b3ba`](https://github.com/apache/spark/commit/053b3ba1b7a84d6a4b355a865f4741935208d978). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501183470 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106410/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501183456 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
cloud-fan commented on a change in pull request #24806: [WIP][SPARK-27856][SQL]
Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292767210
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DecimalType.scala
##
@@ -89,6 +90,7 @@ case class DecimalType(precision: Int, scale: Int) extends
FractionalType {
(precision - scale) <= (dt.precision - dt.scale) && scale <= dt.scale
case dt: IntegralType =>
isTighterThan(DecimalType.forType(dt))
+// For DoubleType/FloatType, the value can be NaN, PositiveInfinity or
NegativeInfinity.
Review comment:
For `isWiderThan`, yes it's not safe to cast float/double to decimal because
of NaN stuff.
For `isTighterThan`, I think it's safe to cast decimal to float/double if
the precision doesn't exceed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
Ngone51 commented on a change in pull request #24841: [SPARK-27369][CORE] Setup
resources when Standalone Worker starts up
URL: https://github.com/apache/spark/pull/24841#discussion_r292767399
##
File path: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
##
@@ -220,6 +225,38 @@ private[deploy] class Worker(
metricsSystem.getServletHandlers.foreach(webUi.attachHandler)
}
+ // TODO if we're starting up multi workers under the same host, discovery
script won't work.
+ private def setupWorkerResources(): Unit = {
+try {
+ resources = resourceFile.map { rFile =>
+ResourceDiscoverer.parseAllocatedFromJsonFile(rFile)
+ }.getOrElse {
+if (resourceDiscoveryScript.isEmpty) {
Review comment:
Not sure what you actually mean here. Just explain my understanding.
`resourceDiscoveryScript` is configured from
`SPARK_WORKER_RESOURCE_DISCOVERY_SCRIPT` or `--resource-script`, which are
specially for Worker. And, application(driver/executor) discoveryScipt config
haven't setup while Worker is starting up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases
AmplabJenkins commented on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases URL: https://github.com/apache/spark/pull/24842#issuecomment-501146872 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases
AmplabJenkins commented on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases URL: https://github.com/apache/spark/pull/24842#issuecomment-501146877 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11650/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501155670 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501155675 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11655/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501155670 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501155675 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11655/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
SparkQA commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501156295 **[Test build #106410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106410/testReport)** for PR 24735 at commit [`053b3ba`](https://github.com/apache/spark/commit/053b3ba1b7a84d6a4b355a865f4741935208d978). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24741: [SPARK-27322][SQL] DataSourceV2 table relation
cloud-fan commented on a change in pull request #24741: [SPARK-27322][SQL]
DataSourceV2 table relation
URL: https://github.com/apache/spark/pull/24741#discussion_r292784645
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -657,6 +659,18 @@ class Analyzer(
}
}
+ /**
+ * Resolve table relations with concrete relations from v2 catalog.
+ */
+ case class ResolveTables(lookupCatalog: LookupCatalog) extends
Rule[LogicalPlan] {
Review comment:
where do we use `lookupCatalog` in this rule?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog
HeartSaVioR commented on issue #23850: [SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog URL: https://github.com/apache/spark/pull/23850#issuecomment-501176314 @dongjoon-hyun Thanks for taking a look at the patch. > If CompactibleFileStreamLog calls purge only when isCompactionBatch returns true, does purge fail in that case? Let me clear the issue - the condition which breaks internal state is, batches to purge contain the latest compaction batch, as further batches will refer the compaction batch. I've described alternatives as well, so please take a look at previous comment: https://github.com/apache/spark/pull/23850#issuecomment-465861957 Btw, even we could purge batches earlier than latest compaction batch, CompactibleFileStreamLog also does the clean up in `deleteExpiredLog` so it is actually not needed. (I'd like to let CompactibleFileStreamLog be responsible to take care about logs by itself.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #23576: [SPARK-26655] [SS] Support multiple aggregates in append mode
HeartSaVioR commented on issue #23576: [SPARK-26655] [SS] Support multiple aggregates in append mode URL: https://github.com/apache/spark/pull/23576#issuecomment-501180447 I just found the case I missed somehow - watermark for (N+1) level of stage shouldn't refer "input" watermark of N level of stage. It should refer "output" watermark of N level of stage. The option 2 doesn't address the input/output watermark hence it should be considered as well. (I just added comment on design doc.) Once input watermark and output watermark for each stateful operator are properly adopted, I think it would work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
AmplabJenkins removed a comment on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql URL: https://github.com/apache/spark/pull/24850#issuecomment-501184366 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
AmplabJenkins removed a comment on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql URL: https://github.com/apache/spark/pull/24850#issuecomment-501184382 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11657/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float
gengliangwang closed pull request #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float URL: https://github.com/apache/spark/pull/24849 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
SparkQA commented on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-501194767 **[Test build #106408 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106408/testReport)** for PR 24806 at commit [`3e29491`](https://github.com/apache/spark/commit/3e2949131a8a7579149a7dd4153650461f5b5da2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
AmplabJenkins commented on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-501194994 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
AmplabJenkins commented on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-501195003 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106408/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
SparkQA removed a comment on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-501149567 **[Test build #106408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106408/testReport)** for PR 24806 at commit [`3e29491`](https://github.com/apache/spark/commit/3e2949131a8a7579149a7dd4153650461f5b5da2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
AmplabJenkins removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501194304 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
AmplabJenkins removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501194318 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106406/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
cloud-fan commented on a change in pull request #24068: [SPARK-27105][SQL]
Optimize away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#discussion_r292759584
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala
##
@@ -362,6 +394,13 @@ object FilterPushdownBenchmark extends BenchmarkBase with
SQLHelper {
}
runBenchmark(s"Pushdown benchmark with many filters") {
+ // This benchmark and the next one are similar in that they both test
predicate pushdown
+ // where the filter itself is very large. There have been cases where
the filter conversion
+ // would take minutes to hours for large filters due to it being
implemented with exponential
+ // complexity in the height of the filter tree.
+ // The difference between these two benchmarks is that this one
benchmarks pushdown with a
+ // large string filter (`a AND b AND c ...`), whereas the next one
benchmarks pushdown with
+ // a large Column-based filter (`col(a) || (col(b) || (col(c)...))`).
Review comment:
If I read this comment correctly, it seems that we should just remove the
next benchmark, as string filter and Column-based filter have no difference
regarding performance. Is there any other critical difference that I missed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
cloud-fan commented on a change in pull request #24806: [WIP][SPARK-27856][SQL]
Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292768996
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -128,8 +128,10 @@ object Cast {
case _ if from == to => true
case (from: NumericType, to: DecimalType) if to.isWiderThan(from) => true
case (from: DecimalType, to: NumericType) if from.isTighterThan(to) => true
-case (f, t) if legalNumericPrecedence(f, t) => true
+case (f: NumericType, t: NumericType) if legalNumericPrecedence(f, t) =>
true
+
case (DateType, TimestampType) => true
+case (NullType, _) => false
Review comment:
Why can't we upcast null to other nullable types? I think it's pretty to
write `INSERT INTO tbl VALUES (1, null)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float
SparkQA commented on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float URL: https://github.com/apache/spark/pull/24849#issuecomment-501168549 **[Test build #106411 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106411/testReport)** for PR 24849 at commit [`44b9fef`](https://github.com/apache/spark/commit/44b9fef2504d9d16740aaf247d4ed64848937710). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float
AmplabJenkins removed a comment on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float URL: https://github.com/apache/spark/pull/24849#issuecomment-501167823 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float
AmplabJenkins removed a comment on issue #24849: [SPARK-28018][SQL] Allow upcasting decimal to double/float URL: https://github.com/apache/spark/pull/24849#issuecomment-501167829 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11656/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501183456 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
SparkQA removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501156295 **[Test build #106410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106410/testReport)** for PR 24735 at commit [`053b3ba`](https://github.com/apache/spark/commit/053b3ba1b7a84d6a4b355a865f4741935208d978). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-501189948 As discussed in https://github.com/apache/spark/pull/24849 , we can't upcast decimal to double/float. I think we should close this one. What do you think? @cloud-fan @rdblue This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
SparkQA removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501149570 **[Test build #106406 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106406/testReport)** for PR 24847 at commit [`fe8f5b6`](https://github.com/apache/spark/commit/fe8f5b6091f11248f00f9231ac926fc675ce8f9b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501194318 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106406/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501194304 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24752: [SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files
cloud-fan commented on a change in pull request #24752:
[SPARK-27893][SQL][PYTHON] Create an integrated test base for Python, Scalar
Pandas, Scala UDF by sql files
URL: https://github.com/apache/spark/pull/24752#discussion_r292761868
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.file.{Files, Paths}
+
+import scala.collection.JavaConverters._
+import scala.util.Try
+
+import org.apache.spark.TestUtils
+import org.apache.spark.api.python.{PythonBroadcast, PythonEvalType,
PythonFunction}
+import org.apache.spark.broadcast.Broadcast
+import org.apache.spark.internal.config.Tests
+import org.apache.spark.sql.catalyst.plans.SQLHelper
+import org.apache.spark.sql.execution.python.UserDefinedPythonFunction
+import org.apache.spark.sql.expressions.SparkUserDefinedFunction
+import org.apache.spark.sql.types.StringType
+
+/**
+ * This object targets to integrate various UDF test cases so that Scalar UDF,
Python UDF and
+ * Scalar Pandas UDFs can be tested in SBT & Maven tests.
+ *
+ * The available UDFs cast input to strings, which take one column as input
and return a string
+ * type column as output.
+ *
+ * To register Scala UDF in SQL:
+ * {{{
+ * registerTestUDF(TestScalaUDF(name = "udf_name"), spark)
+ * }}}
+ *
+ * To register Python UDF in SQL:
+ * {{{
+ * registerTestUDF(TestPythonUDF(name = "udf_name"), spark)
+ * }}}
+ *
+ * To register Scalar Pandas UDF in SQL:
+ * {{{
+ * registerTestUDF(TestScalarPandasUDF(name = "udf_name"), spark)
+ * }}}
+ *
+ * To use it in Scala API and SQL:
+ * {{{
+ * sql("SELECT udf_name(1)")
+ * spark.select(expr("udf_name(1)")
+ * }}}
+ */
+object IntegratedUDFTestUtils extends SQLHelper {
+ import scala.sys.process._
+
+ private lazy val pythonPath = sys.env.getOrElse("PYTHONPATH", "")
+ private lazy val sparkHome = if (sys.props.contains(Tests.IS_TESTING.key)) {
+assert(sys.props.contains("spark.test.home"), "spark.test.home is not
set.")
+sys.props("spark.test.home")
+ } else {
+assert(sys.env.contains("SPARK_HOME"), "SPARK_HOME is not set.")
+sys.env("SPARK_HOME")
+ }
+ // Note that we will directly refer pyspark's source, not the zip from a
regular build.
+ // It is possible the test is being ran without the build.
+ private lazy val sourcePath = Paths.get(sparkHome, "python").toAbsolutePath
+ private lazy val py4jPath = Paths.get(
+sparkHome, "python", "lib", "py4j-0.10.8.1-src.zip").toAbsolutePath
Review comment:
do we need to update this when we upgrade py4j in Spark?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
cloud-fan commented on a change in pull request #24735: [SPARK-27871][SQL]
LambdaVariable should use per-query unique IDs instead of globally unique IDs
URL: https://github.com/apache/spark/pull/24735#discussion_r292763275
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
##
@@ -228,3 +228,31 @@ object ObjectSerializerPruning extends Rule[LogicalPlan] {
}
}
}
+
+/**
+ * Reassigns per-query unique IDs to `LambdaVariable`s, whose original IDs are
globally unique. This
+ * can help Spark to hit codegen cache more often and improve performance.
+ */
+object ReassignLambdaVariableID extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+if (!SQLConf.get.getConf(SQLConf.OPTIMIZER_REASSIGN_LAMBDA_VARIABLE_ID))
return plan
+
+// The original LambdaVariable IDs are all positive. To avoid conflicts,
the new IDs are all
+// negative and starts with -1.
+var newId = -1L
+val oldIdToNewId = scala.collection.mutable.Map.empty[Long, Long]
+
+plan.transformAllExpressions {
Review comment:
fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
AmplabJenkins removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501148103 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
SparkQA removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501141551 **[Test build #106404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106404/testReport)** for PR 24735 at commit [`053b3ba`](https://github.com/apache/spark/commit/053b3ba1b7a84d6a4b355a865f4741935208d978). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501148103 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501148059 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
SparkQA removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501125135 **[Test build #106402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106402/testReport)** for PR 24829 at commit [`0a425c4`](https://github.com/apache/spark/commit/0a425c41b26225512cb9d0e8cb58986d76513f6c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
dongjoon-hyun commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501148006 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501147895 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
SparkQA removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501130937 **[Test build #106403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106403/testReport)** for PR 24847 at commit [`fe8f5b6`](https://github.com/apache/spark/commit/fe8f5b6091f11248f00f9231ac926fc675ce8f9b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501148059 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501148069 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106402/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases
AmplabJenkins removed a comment on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases URL: https://github.com/apache/spark/pull/24842#issuecomment-501147916 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501148106 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106403/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
cloud-fan commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501155225 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
wangyum commented on a change in pull request #24850:
[WIP][SPARK-28020][SQL][TEST] Port date.sql
URL: https://github.com/apache/spark/pull/24850#discussion_r292811558
##
File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql
##
@@ -0,0 +1,358 @@
+--
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+--
+-- DATE
+--
https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/date.sql
+
+CREATE TABLE DATE_TBL (f1 date) USING parquet;
+
+INSERT INTO DATE_TBL VALUES ('1957-04-09');
+INSERT INTO DATE_TBL VALUES ('1957-06-13');
+INSERT INTO DATE_TBL VALUES ('1996-02-28');
+INSERT INTO DATE_TBL VALUES ('1996-02-29');
+INSERT INTO DATE_TBL VALUES ('1996-03-01');
+INSERT INTO DATE_TBL VALUES ('1996-03-02');
+INSERT INTO DATE_TBL VALUES ('1997-02-28');
+-- [SPARK-27923] Skip invalid date: 1997-02-29
+-- INSERT INTO DATE_TBL VALUES ('1997-02-29');
+INSERT INTO DATE_TBL VALUES ('1997-03-01');
+INSERT INTO DATE_TBL VALUES ('1997-03-02');
+INSERT INTO DATE_TBL VALUES ('2000-04-01');
+INSERT INTO DATE_TBL VALUES ('2000-04-02');
+INSERT INTO DATE_TBL VALUES ('2000-04-03');
+INSERT INTO DATE_TBL VALUES ('2038-04-08');
+INSERT INTO DATE_TBL VALUES ('2039-04-09');
+INSERT INTO DATE_TBL VALUES ('2040-04-10');
+
+SELECT f1 AS `Fifteen` FROM DATE_TBL;
+
+SELECT f1 AS `Nine` FROM DATE_TBL WHERE f1 < '2000-01-01';
+
+SELECT f1 AS `Three` FROM DATE_TBL
+ WHERE f1 BETWEEN '2000-01-01' AND '2001-01-01';
+
+-- Skip the formats that we do not supported. Please check [SPARK-8995] for
all supported formats
+--
+-- Check all the documented input formats
+--
+-- SET datestyle TO iso; -- display results in ISO
+
+-- SET datestyle TO ymd;
+
+-- SELECT date 'January 8, 1999';
+SELECT date '1999-01-08';
+SELECT date '1999-01-18';
+-- SELECT date '1/8/1999';
+-- SELECT date '1/18/1999';
+-- SELECT date '18/1/1999';
+-- SELECT date '01/02/03';
+-- SELECT date '19990108';
+-- SELECT date '990108';
+-- SELECT date '1999.008';
+-- SELECT date 'J2451187';
+-- SELECT date 'January 8, 99 BC';
+
+-- SELECT date '99-Jan-08';
+-- SELECT date '1999-Jan-08';
+-- SELECT date '08-Jan-99';
+-- SELECT date '08-Jan-1999';
+-- SELECT date 'Jan-08-99';
+-- SELECT date 'Jan-08-1999';
+-- SELECT date '99-08-Jan';
+-- SELECT date '1999-08-Jan';
+
+-- SELECT date '99 Jan 08';
+SELECT date '1999 Jan 08';
+-- SELECT date '08 Jan 99';
+-- SELECT date '08 Jan 1999';
+-- SELECT date 'Jan 08 99';
+-- SELECT date 'Jan 08 1999';
+-- SELECT date '99 08 Jan';
+SELECT date '1999 08 Jan';
+
+-- SELECT date '99-01-08';
+SELECT date '1999-01-08';
+-- SELECT date '08-01-99';
+-- SELECT date '08-01-1999';
+-- SELECT date '01-08-99';
+-- SELECT date '01-08-1999';
+-- SELECT date '99-08-01';
+SELECT date '1999-08-01';
+
+-- SELECT date '99 01 08';
+SELECT date '1999 01 08';
+-- SELECT date '08 01 99';
+-- SELECT date '08 01 1999';
+-- SELECT date '01 08 99';
+-- SELECT date '01 08 1999';
+-- SELECT date '99 08 01';
+SELECT date '1999 08 01';
+
+-- SET datestyle TO dmy;
+
+-- SELECT date 'January 8, 1999';
+SELECT date '1999-01-08';
+-- SELECT date '1999-01-18';
+-- SELECT date '1/8/1999';
+-- SELECT date '1/18/1999';
+-- SELECT date '18/1/1999';
+-- SELECT date '01/02/03';
+-- SELECT date '19990108';
+-- SELECT date '990108';
+-- SELECT date '1999.008';
+-- SELECT date 'J2451187';
+-- SELECT date 'January 8, 99 BC';
+
+-- SELECT date '99-Jan-08';
+-- SELECT date '1999-Jan-08';
+-- SELECT date '08-Jan-99';
+-- SELECT date '08-Jan-1999';
+-- SELECT date 'Jan-08-99';
+-- SELECT date 'Jan-08-1999';
+-- SELECT date '99-08-Jan';
+-- SELECT date '1999-08-Jan';
+
+-- SELECT date '99 Jan 08';
+SELECT date '1999 Jan 08';
+-- SELECT date '08 Jan 99';
+-- SELECT date '08 Jan 1999';
+-- SELECT date 'Jan 08 99';
+-- SELECT date 'Jan 08 1999';
+-- SELECT date '99 08 Jan';
+SELECT date '1999 08 Jan';
+
+-- SELECT date '99-01-08';
+SELECT date '1999-01-08';
+-- SELECT date '08-01-99';
+-- SELECT date '08-01-1999';
+-- SELECT date '01-08-99';
+-- SELECT date '01-08-1999';
+-- SELECT date '99-08-01';
+SELECT date '1999-08-01';
+
+-- SELECT date '99 01 08';
+SELECT date '1999 01 08';
+-- SELECT date '08 01 99';
+-- SELECT date '08 01 1999';
+-- SELECT date '01 08 99';
+-- SELECT date '01 08 1999';
+-- SELECT date '99 08 01';
+SELECT date '1999 08 01';
+
+-- SET datestyle TO mdy;
+
+-- SELECT date 'January 8, 1999';
+SELECT date '1999-01-08';
+SELECT date '1999-01-18';
+-- SELECT date '1/8/1999';
+-- SELECT date '1/18/1999';
+-- SELECT date '18/1/1999';
+-- SELECT date '01/02/03';
+-- SELECT date '19990108';
+-- SELECT date '990108';
+-- SELECT date '1999.008';
+-- SELECT date 'J2451187';
+-- SELECT date 'January 8, 99 BC';
+
+-- SELECT date '99-Jan-08';
+-- SELECT date '1999-Jan-08';
+-- SELECT date '08-Jan-99';
+-- SELECT date '08-Jan-1999';
+-- SELECT date 'Jan-08-99';
+-- SELECT date 'Jan-08-1999';
+-- SELECT date '99-08-Jan';
+-- SELECT date '1999-08-Jan';
+
+-- SELECT date '99 Jan 08';
+SELECT date '1999 Jan
[GitHub] [spark] AmplabJenkins commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
AmplabJenkins commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql URL: https://github.com/apache/spark/pull/24850#issuecomment-501184366 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
AmplabJenkins commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql URL: https://github.com/apache/spark/pull/24850#issuecomment-501184382 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11657/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501183470 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106410/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
wangyum commented on a change in pull request #24850:
[WIP][SPARK-28020][SQL][TEST] Port date.sql
URL: https://github.com/apache/spark/pull/24850#discussion_r292811714
##
File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql
##
@@ -0,0 +1,358 @@
+--
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+--
+-- DATE
+--
https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/date.sql
+
+CREATE TABLE DATE_TBL (f1 date) USING parquet;
+
+INSERT INTO DATE_TBL VALUES ('1957-04-09');
+INSERT INTO DATE_TBL VALUES ('1957-06-13');
+INSERT INTO DATE_TBL VALUES ('1996-02-28');
+INSERT INTO DATE_TBL VALUES ('1996-02-29');
+INSERT INTO DATE_TBL VALUES ('1996-03-01');
+INSERT INTO DATE_TBL VALUES ('1996-03-02');
+INSERT INTO DATE_TBL VALUES ('1997-02-28');
+-- [SPARK-27923] Skip invalid date: 1997-02-29
+-- INSERT INTO DATE_TBL VALUES ('1997-02-29');
+INSERT INTO DATE_TBL VALUES ('1997-03-01');
+INSERT INTO DATE_TBL VALUES ('1997-03-02');
+INSERT INTO DATE_TBL VALUES ('2000-04-01');
+INSERT INTO DATE_TBL VALUES ('2000-04-02');
+INSERT INTO DATE_TBL VALUES ('2000-04-03');
+INSERT INTO DATE_TBL VALUES ('2038-04-08');
+INSERT INTO DATE_TBL VALUES ('2039-04-09');
+INSERT INTO DATE_TBL VALUES ('2040-04-10');
+
+SELECT f1 AS `Fifteen` FROM DATE_TBL;
+
+SELECT f1 AS `Nine` FROM DATE_TBL WHERE f1 < '2000-01-01';
+
+SELECT f1 AS `Three` FROM DATE_TBL
+ WHERE f1 BETWEEN '2000-01-01' AND '2001-01-01';
+
+-- Skip the formats that we do not supported. Please check [SPARK-8995] for
all supported formats
+--
+-- Check all the documented input formats
+--
+-- SET datestyle TO iso; -- display results in ISO
+
+-- SET datestyle TO ymd;
+
+-- SELECT date 'January 8, 1999';
+SELECT date '1999-01-08';
+SELECT date '1999-01-18';
+-- SELECT date '1/8/1999';
+-- SELECT date '1/18/1999';
+-- SELECT date '18/1/1999';
+-- SELECT date '01/02/03';
+-- SELECT date '19990108';
+-- SELECT date '990108';
+-- SELECT date '1999.008';
+-- SELECT date 'J2451187';
+-- SELECT date 'January 8, 99 BC';
+
+-- SELECT date '99-Jan-08';
+-- SELECT date '1999-Jan-08';
+-- SELECT date '08-Jan-99';
+-- SELECT date '08-Jan-1999';
+-- SELECT date 'Jan-08-99';
+-- SELECT date 'Jan-08-1999';
+-- SELECT date '99-08-Jan';
+-- SELECT date '1999-08-Jan';
+
+-- SELECT date '99 Jan 08';
+SELECT date '1999 Jan 08';
+-- SELECT date '08 Jan 99';
+-- SELECT date '08 Jan 1999';
+-- SELECT date 'Jan 08 99';
+-- SELECT date 'Jan 08 1999';
+-- SELECT date '99 08 Jan';
+SELECT date '1999 08 Jan';
+
+-- SELECT date '99-01-08';
+SELECT date '1999-01-08';
+-- SELECT date '08-01-99';
+-- SELECT date '08-01-1999';
+-- SELECT date '01-08-99';
+-- SELECT date '01-08-1999';
+-- SELECT date '99-08-01';
+SELECT date '1999-08-01';
+
+-- SELECT date '99 01 08';
+SELECT date '1999 01 08';
+-- SELECT date '08 01 99';
+-- SELECT date '08 01 1999';
+-- SELECT date '01 08 99';
+-- SELECT date '01 08 1999';
+-- SELECT date '99 08 01';
+SELECT date '1999 08 01';
+
+-- SET datestyle TO dmy;
+
+-- SELECT date 'January 8, 1999';
+SELECT date '1999-01-08';
+-- SELECT date '1999-01-18';
+-- SELECT date '1/8/1999';
+-- SELECT date '1/18/1999';
+-- SELECT date '18/1/1999';
+-- SELECT date '01/02/03';
+-- SELECT date '19990108';
+-- SELECT date '990108';
+-- SELECT date '1999.008';
+-- SELECT date 'J2451187';
+-- SELECT date 'January 8, 99 BC';
+
+-- SELECT date '99-Jan-08';
+-- SELECT date '1999-Jan-08';
+-- SELECT date '08-Jan-99';
+-- SELECT date '08-Jan-1999';
+-- SELECT date 'Jan-08-99';
+-- SELECT date 'Jan-08-1999';
+-- SELECT date '99-08-Jan';
+-- SELECT date '1999-08-Jan';
+
+-- SELECT date '99 Jan 08';
+SELECT date '1999 Jan 08';
+-- SELECT date '08 Jan 99';
+-- SELECT date '08 Jan 1999';
+-- SELECT date 'Jan 08 99';
+-- SELECT date 'Jan 08 1999';
+-- SELECT date '99 08 Jan';
+SELECT date '1999 08 Jan';
+
+-- SELECT date '99-01-08';
+SELECT date '1999-01-08';
+-- SELECT date '08-01-99';
+-- SELECT date '08-01-1999';
+-- SELECT date '01-08-99';
+-- SELECT date '01-08-1999';
+-- SELECT date '99-08-01';
+SELECT date '1999-08-01';
+
+-- SELECT date '99 01 08';
+SELECT date '1999 01 08';
+-- SELECT date '08 01 99';
+-- SELECT date '08 01 1999';
+-- SELECT date '01 08 99';
+-- SELECT date '01 08 1999';
+-- SELECT date '99 08 01';
+SELECT date '1999 08 01';
+
+-- SET datestyle TO mdy;
+
+-- SELECT date 'January 8, 1999';
+SELECT date '1999-01-08';
+SELECT date '1999-01-18';
+-- SELECT date '1/8/1999';
+-- SELECT date '1/18/1999';
+-- SELECT date '18/1/1999';
+-- SELECT date '01/02/03';
+-- SELECT date '19990108';
+-- SELECT date '990108';
+-- SELECT date '1999.008';
+-- SELECT date 'J2451187';
+-- SELECT date 'January 8, 99 BC';
+
+-- SELECT date '99-Jan-08';
+-- SELECT date '1999-Jan-08';
+-- SELECT date '08-Jan-99';
+-- SELECT date '08-Jan-1999';
+-- SELECT date 'Jan-08-99';
+-- SELECT date 'Jan-08-1999';
+-- SELECT date '99-08-Jan';
+-- SELECT date '1999-08-Jan';
+
+-- SELECT date '99 Jan 08';
+SELECT date '1999 Jan
[GitHub] [spark] zuotingbing commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed.
zuotingbing commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed. URL: https://github.com/apache/spark/pull/24848#issuecomment-501195379 cc @vanzin Could you please take a review? thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
AmplabJenkins removed a comment on issue #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-501195003 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106408/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
SparkQA commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501138979 **[Test build #106401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106401/testReport)** for PR 24792 at commit [`a912b87`](https://github.com/apache/spark/commit/a912b87893d22e603d334429b3ca3de644ca2780). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins removed a comment on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501151129 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
AmplabJenkins removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501151117 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
AmplabJenkins removed a comment on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501151122 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11653/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501151129 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
AmplabJenkins commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501151138 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11654/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501151117 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
AmplabJenkins commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501151122 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11653/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed.
AmplabJenkins commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed. URL: https://github.com/apache/spark/pull/24848#issuecomment-501163279 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on issue #21613: [SPARK-24629][SQL]thrift server memory leaks when Beeline session is closed
LantaoJin commented on issue #21613: [SPARK-24629][SQL]thrift server memory leaks when Beeline session is closed URL: https://github.com/apache/spark/pull/21613#issuecomment-501169113 > I'm thinking if this issue can be solved together with [SPARK-26533](https://issues.apache.org/jira/browse/SPARK-26533)? Not yet PR opened from SPARK-26533 and this bug should be fixed first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806:
[WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292794102
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##
@@ -356,8 +358,28 @@ case class PreprocessTableInsertion(conf: SQLConf)
extends Rule[LogicalPlan] {
s"including ${staticPartCols.size} partition column(s) having
constant value(s).")
}
-val newQuery = DDLPreprocessingUtils.castAndRenameQueryOutput(
- insert.query, expectedColumns, conf)
+val newQuery = if
(conf.getConf(SQLConf.LEGACY_INSERT_TABLE_TYPE_COERCION)) {
+ DDLPreprocessingUtils.castAndRenameQueryOutput(insert.query,
expectedColumns, conf)
+} else {
+ val errors = new mutable.ArrayBuffer[String]()
Review comment:
We need to figure out if we can up-cast decimal to double/float.
If we can't, then maybe we can't continue this PR.
I have created a new PR https://github.com/apache/spark/pull/24849
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806:
[WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292794102
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##
@@ -356,8 +358,28 @@ case class PreprocessTableInsertion(conf: SQLConf)
extends Rule[LogicalPlan] {
s"including ${staticPartCols.size} partition column(s) having
constant value(s).")
}
-val newQuery = DDLPreprocessingUtils.castAndRenameQueryOutput(
- insert.query, expectedColumns, conf)
+val newQuery = if
(conf.getConf(SQLConf.LEGACY_INSERT_TABLE_TYPE_COERCION)) {
+ DDLPreprocessingUtils.castAndRenameQueryOutput(insert.query,
expectedColumns, conf)
+} else {
+ val errors = new mutable.ArrayBuffer[String]()
Review comment:
We need to figure out if we can up-cast decimal to double/float.
If we can't, then maybe we can't continue this PR.
I have created a new PR for the upcasting:
https://github.com/apache/spark/pull/24849
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql
SparkQA commented on issue #24850: [WIP][SPARK-28020][SQL][TEST] Port date.sql URL: https://github.com/apache/spark/pull/24850#issuecomment-501185342 **[Test build #106413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106413/testReport)** for PR 24850 at commit [`69470df`](https://github.com/apache/spark/commit/69470df00127fc9f1373d33fae90e8bd9d01dfa0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases
SparkQA removed a comment on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases URL: https://github.com/apache/spark/pull/24842#issuecomment-501145319 **[Test build #106405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106405/testReport)** for PR 24842 at commit [`92cc5d2`](https://github.com/apache/spark/commit/92cc5d2ee59871278d3042825f7b96e17a86897c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases
AmplabJenkins commented on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases URL: https://github.com/apache/spark/pull/24842#issuecomment-501147916 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806:
[WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292770658
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DecimalType.scala
##
@@ -89,6 +90,7 @@ case class DecimalType(precision: Int, scale: Int) extends
FractionalType {
(precision - scale) <= (dt.precision - dt.scale) && scale <= dt.scale
case dt: IntegralType =>
isTighterThan(DecimalType.forType(dt))
+// For DoubleType/FloatType, the value can be NaN, PositiveInfinity or
NegativeInfinity.
Review comment:
> For isTighterThan, I think it's safe to cast decimal to float/double if
the precision doesn't exceed?
Yes, it is. I was about to push the commit to fix tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1
SparkQA commented on issue #24847: [SPARK-28013][BUILD][SS] Upgrade to Kafka 2.2.1 URL: https://github.com/apache/spark/pull/24847#issuecomment-501147860 **[Test build #106403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106403/testReport)** for PR 24847 at commit [`fe8f5b6`](https://github.com/apache/spark/commit/fe8f5b6091f11248f00f9231ac926fc675ce8f9b). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501147895 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-501147903 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106404/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] peter-toth edited a comment on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases
peter-toth edited a comment on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases URL: https://github.com/apache/spark/pull/24842#issuecomment-501145852 Thanks @liancheng for the review. I've moved analysis error test cases to `AnalysisSuite` as you suggested. But I left the positive test case in `SQLQuerySuite` as I think the value of `x` column should be tested. Please let me know if you disagree. BTW there is another improvement PR regarding `WITH` clause I opened recently: https://github.com/apache/spark/pull/24831 Any comment is very welcome if you have some time to review it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3]
SparkQA commented on issue #24829: [WIP][SPARK-27988][SQL][TEST] Port AGGREGATES.sql [Part 3] URL: https://github.com/apache/spark/pull/24829#issuecomment-501147856 **[Test build #106402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106402/testReport)** for PR 24829 at commit [`0a425c4`](https://github.com/apache/spark/commit/0a425c41b26225512cb9d0e8cb58986d76513f6c). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases
AmplabJenkins commented on issue #24842: [SPARK-28002][SQL] Support WITH clause column aliases URL: https://github.com/apache/spark/pull/24842#issuecomment-501147926 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106405/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
