[GitHub] [spark] SparkQA commented on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
SparkQA commented on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp URL: https://github.com/apache/spark/pull/27906#issuecomment-599892215 **[Test build #119914 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119914/testReport)** for PR 27906 at commit [`27095da`](https://github.com/apache/spark/commit/27095da963d70fca278d558b8e877e53843087eb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393457826
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -235,7 +237,7 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
}
test("SPARK-30958: parse timestamp with negative year") {
-val formatter1 = TimestampFormatter("-MM-dd HH:mm:ss", ZoneOffset.UTC)
+val formatter1 = TimestampFormatter("-MM-dd HH:mm:ss", ZoneOffset.UTC,
true)
Review comment:
yes, the actual parser should work with patterns w/o SSS too. I don't add
new tests in TimestampFormatterSuite but modify the existing ones because we
have e2e enough tests in `date_time.sql`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393457826
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -235,7 +237,7 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
}
test("SPARK-30958: parse timestamp with negative year") {
-val formatter1 = TimestampFormatter("-MM-dd HH:mm:ss", ZoneOffset.UTC)
+val formatter1 = TimestampFormatter("-MM-dd HH:mm:ss", ZoneOffset.UTC,
true)
Review comment:
yes, the actual parser should work with patterns w/o SSS too. I don't add
new tests in TimestampFormatterSuite but modify the existing ones because we
have enough e2e tests in `date_time.sql`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393457826
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -235,7 +237,7 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
}
test("SPARK-30958: parse timestamp with negative year") {
-val formatter1 = TimestampFormatter("-MM-dd HH:mm:ss", ZoneOffset.UTC)
+val formatter1 = TimestampFormatter("-MM-dd HH:mm:ss", ZoneOffset.UTC,
true)
Review comment:
yes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.wareho
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27933#issuecomment-599890532 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24638/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.wareho
AmplabJenkins removed a comment on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27933#issuecomment-599890525 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27933#issuecomment-599890532 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24638/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
AmplabJenkins commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27933#issuecomment-599890525 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
SparkQA commented on issue #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27933#issuecomment-599890184 **[Test build #119913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119913/testReport)** for PR 27933 at commit [`13a3fe8`](https://github.com/apache/spark/commit/13a3fe80cbcdde4b73b6829a5421be46e7c022a9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393457332
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -64,7 +65,8 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
DateTimeTestUtils.outstandingTimezonesIds.foreach { zoneId =>
val formatter = TimestampFormatter(
"-MM-dd'T'HH:mm:ss.SS",
-DateTimeUtils.getZoneId(zoneId))
+DateTimeUtils.getZoneId(zoneId),
+needVarLengthSecondFraction = true)
Review comment:
oh. I mean this should be false too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393457355
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -235,7 +237,7 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
}
test("SPARK-30958: parse timestamp with negative year") {
-val formatter1 = TimestampFormatter("-MM-dd HH:mm:ss", ZoneOffset.UTC)
+val formatter1 = TimestampFormatter("-MM-dd HH:mm:ss", ZoneOffset.UTC,
true)
Review comment:
is the point to test the actual parser? with `needVarLengthSecondFraction =
true`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27920: [SPARK-31102][SQL] Spark-sql fails to parse when contains comment.
maropu commented on a change in pull request #27920: [SPARK-31102][SQL]
Spark-sql fails to parse when contains comment.
URL: https://github.com/apache/spark/pull/27920#discussion_r393456400
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##
@@ -402,24 +402,14 @@ class CliSuite extends SparkFunSuite with
BeforeAndAfterAll with Logging {
}
test("SPARK-30049 Should not complain for quotes in commented lines") {
-runCliWithin(1.minute)(
+runCliWithin(3.minute)(
"""SELECT concat('test', 'comment') -- someone's comment here
|;""".stripMargin -> "testcomment"
)
- }
-
- test("SPARK-30049 Should not complain for quotes in commented with
multi-lines") {
-runCliWithin(1.minute)(
- """SELECT concat('test', 'comment') -- someone's comment here \\
-| comment continues here with single ' quote \\
-| extra ' \\
-|;""".stripMargin -> "testcomment"
-)
-runCliWithin(1.minute)(
- """SELECT concat('test', 'comment') -- someone's comment here \\
-| comment continues here with single ' quote \\
-| extra ' \\
-| ;""".stripMargin -> "testcomment"
Review comment:
If we can, the fix in `SqlBase.g4` (`SIMPLE_COMENT`) looks fine to me and I
think the queries above should work in Spark SQL:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4#L1811
Could you try?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27920: [SPARK-31102][SQL] Spark-sql fails to parse when contains comment.
maropu commented on a change in pull request #27920: [SPARK-31102][SQL]
Spark-sql fails to parse when contains comment.
URL: https://github.com/apache/spark/pull/27920#discussion_r393456400
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##
@@ -402,24 +402,14 @@ class CliSuite extends SparkFunSuite with
BeforeAndAfterAll with Logging {
}
test("SPARK-30049 Should not complain for quotes in commented lines") {
-runCliWithin(1.minute)(
+runCliWithin(3.minute)(
"""SELECT concat('test', 'comment') -- someone's comment here
|;""".stripMargin -> "testcomment"
)
- }
-
- test("SPARK-30049 Should not complain for quotes in commented with
multi-lines") {
-runCliWithin(1.minute)(
- """SELECT concat('test', 'comment') -- someone's comment here \\
-| comment continues here with single ' quote \\
-| extra ' \\
-|;""".stripMargin -> "testcomment"
-)
-runCliWithin(1.minute)(
- """SELECT concat('test', 'comment') -- someone's comment here \\
-| comment continues here with single ' quote \\
-| extra ' \\
-| ;""".stripMargin -> "testcomment"
Review comment:
If we can, the fix in `SqlBase.g4` (`SIMPLE_COMENT`) looks fine to me:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4#L1811
Could you try?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393456155
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -83,9 +85,9 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
2177456523456789L,
11858049903010203L).foreach { micros =>
DateTimeTestUtils.outstandingZoneIds.foreach { zoneId =>
- val formatter = TimestampFormatter(pattern, zoneId)
- val timestamp = formatter.format(micros)
- val parsed = formatter.parse(timestamp)
+ val timestamp = TimestampFormatter(pattern, zoneId).format(micros)
+ val parsed = TimestampFormatter(
+pattern, zoneId, needVarLengthSecondFraction =
true).parse(timestamp)
Review comment:
ok makes sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393456067
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -64,7 +65,8 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
DateTimeTestUtils.outstandingTimezonesIds.foreach { zoneId =>
val formatter = TimestampFormatter(
"-MM-dd'T'HH:mm:ss.SS",
-DateTimeUtils.getZoneId(zoneId))
+DateTimeUtils.getZoneId(zoneId),
+needVarLengthSecondFraction = true)
Review comment:
really? this test is for formatting, and the actual formatting doesn't do
padding, so this flag should be false.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-599888705 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-599888710 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24637/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-599888705 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn opened a new pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.di
yaooqinn opened a new pull request #27933: [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sSpark Cli does not respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27933 ### What changes were proposed in this pull request? In Spark CLI, we create a hive `CliSessionState` and it does not load the `hive-site.xml`. So the configurations in `hive-site.xml` will not take effects like other spark-hive integration apps. Also, the warehouse directory is not correctly picked. If the `default` database does not exist, the `CliSessionState` will create one during the first time it talks to the metastore. The `Location` of the default DB will be neither the value of `spark.sql.warehousr.dir` nor the user-specified value of `hive.metastore.warehourse.dir`, but the default value of `hive.metastore.warehourse.dir `which will always be `/user/hive/warehouse`. ### Why are the changes needed? fix bug for Spark SQL cli to pick right confs ### Does this PR introduce any user-facing change? yes, the non-exists default database will be created in the location specified by the users via `spark.sql.warehouse.dir` or `hive.metastore.warehouse.dir`, or the default value of `spark.sql.warehouse.dir` if none of them specified. ### How was this patch tested? add cli ut This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-599888710 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24637/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
SparkQA commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-599888366 **[Test build #119912 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119912/testReport)** for PR 27897 at commit [`1d56c0e`](https://github.com/apache/spark/commit/1d56c0e4fbe9a26232d6856a253371d16f24d4da). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add
SHOW VIEWS command
URL: https://github.com/apache/spark/pull/27897#discussion_r393454589
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
##
@@ -530,6 +523,16 @@ class ResolveSessionCatalog(
replace,
viewType)
+case ShowViewsStatement(SessionCatalogAndNamespace(_, ns), pattern) =>
+ val namespace = if (ns.isEmpty) {
+None
+ } else if (ns.length == 1) {
+Some(ns.head)
+ } else {
+throw new AnalysisException(s"The database name is not valid:
${ns.quoted}")
Review comment:
Yea, updated in 1d56c0e4fbe9a26232d6856a253371d16f24d4da, add negative test
case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment
AmplabJenkins removed a comment on issue #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment URL: https://github.com/apache/spark/pull/27930#issuecomment-599886895 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment
AmplabJenkins removed a comment on issue #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment URL: https://github.com/apache/spark/pull/27930#issuecomment-599886901 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24636/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add
SHOW VIEWS command
URL: https://github.com/apache/spark/pull/27897#discussion_r393420719
##
File path: docs/sql-ref-syntax-aux-show-views.md
##
@@ -0,0 +1,107 @@
+---
+layout: global
+title: SHOW VIEWS
+displayTitle: SHOW VIEWS
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+### Description
+
+The `SHOW VIEWS` statement returns all the views for an optionally specified
database.
+Additionally, the output of this statement may be filtered by an optional
matching
+pattern. If no database is specified then the views are returned from the
+current database. Note that both global and local temporary views are also
returned.
+
+### Syntax
+{% highlight sql %}
+SHOW VIEWS [ { FROM | IN } database_name ] [ LIKE 'regex_pattern' ]
+{% endhighlight %}
+
+### Parameters
+
+ { FROM | IN } database_name
+
+ Specifies the database name from which views are listed.
+
+ LIKE regex_pattern
+
+ Specifies the regular expression pattern that is used to filter out
unwanted views.
+
+ Except for `*` and `|` character, the pattern works like a
regex.
+ `*` alone matches 0 or more characters and `|` is used to
separate multiple different regexes,
+ any of which can match.
+ The leading and trailing blanks are trimmed in the input
pattern before processing.
+
+
+
+
+
+### Example
+{% highlight sql %}
+-- List all views in default database
Review comment:
I see, updated in dc6e3cb21e5de2a2c591b42014cc03fc4418cca3. Will update
`SHOW TABLES` in follow-up PR. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment
AmplabJenkins commented on issue #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment URL: https://github.com/apache/spark/pull/27930#issuecomment-599886895 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment
AmplabJenkins commented on issue #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment URL: https://github.com/apache/spark/pull/27930#issuecomment-599886901 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24636/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment
SparkQA commented on issue #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment URL: https://github.com/apache/spark/pull/27930#issuecomment-599886622 **[Test build #119910 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119910/testReport)** for PR 27930 at commit [`a1bfa96`](https://github.com/apache/spark/commit/a1bfa9623ab4a5162709bf55d56f055300fd34f2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
maropu commented on a change in pull request #27897: [SPARK-31113][SQL] Add
SHOW VIEWS command
URL: https://github.com/apache/spark/pull/27897#discussion_r393453299
##
File path: docs/sql-ref-syntax-aux-show-views.md
##
@@ -0,0 +1,107 @@
+---
+layout: global
+title: SHOW VIEWS
+displayTitle: SHOW VIEWS
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+### Description
+
+The `SHOW VIEWS` statement returns all the views for an optionally specified
database.
+Additionally, the output of this statement may be filtered by an optional
matching
+pattern. If no database is specified then the views are returned from the
+current database. Note that both global and local temporary views are also
returned.
+
+### Syntax
+{% highlight sql %}
+SHOW VIEWS [ { FROM | IN } database_name ] [ LIKE 'regex_pattern' ]
+{% endhighlight %}
+
+### Parameters
+
+ { FROM | IN } database_name
+
+ Specifies the database name from which views are listed.
+
+ LIKE regex_pattern
+
+ Specifies the regular expression pattern that is used to filter out
unwanted views.
+
+ Except for `*` and `|` character, the pattern works like a
regex.
+ `*` alone matches 0 or more characters and `|` is used to
separate multiple different regexes,
+ any of which can match.
+ The leading and trailing blanks are trimmed in the input
pattern before processing.
+
+
+
+
+
+### Example
+{% highlight sql %}
+-- List all views in default database
Review comment:
Yea, thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27877: [WIP][SPARK-30541][TESTS] Implement KafkaDelegationTokenSuite with testRetry
SparkQA commented on issue #27877: [WIP][SPARK-30541][TESTS] Implement KafkaDelegationTokenSuite with testRetry URL: https://github.com/apache/spark/pull/27877#issuecomment-599886589 **[Test build #119911 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119911/testReport)** for PR 27877 at commit [`bc4fb8b`](https://github.com/apache/spark/commit/bc4fb8bdd37dd64a217126bc6edb48eaee91c454). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment
maropu commented on a change in pull request #27930: [SQL][MINOR] Update the
DataFrameWriter.bucketBy comment
URL: https://github.com/apache/spark/pull/27930#discussion_r393452684
##
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##
@@ -197,8 +197,8 @@ final class DataFrameWriter[T] private[sql](ds:
Dataset[T]) {
}
/**
- * Buckets the output by the given columns. If specified, the output is laid
out on the file
- * system similar to Hive's bucketing scheme.
+ * Buckets the output by the given columns. Note that the output follows a
Spark SQL specific
+ * bucketing scheme based on the Hive scheme.
Review comment:
updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment
maropu commented on a change in pull request #27930: [SQL][MINOR] Update the
DataFrameWriter.bucketBy comment
URL: https://github.com/apache/spark/pull/27930#discussion_r393452207
##
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##
@@ -197,8 +197,8 @@ final class DataFrameWriter[T] private[sql](ds:
Dataset[T]) {
}
/**
- * Buckets the output by the given columns. If specified, the output is laid
out on the file
- * system similar to Hive's bucketing scheme.
+ * Buckets the output by the given columns. Note that the output follows a
Spark SQL specific
+ * bucketing scheme based on the Hive scheme.
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on issue #27877: [WIP][SPARK-30541][TESTS] Implement KafkaDelegationTokenSuite with testRetry
gaborgsomogyi commented on issue #27877: [WIP][SPARK-30541][TESTS] Implement KafkaDelegationTokenSuite with testRetry URL: https://github.com/apache/spark/pull/27877#issuecomment-599885091 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core
AmplabJenkins commented on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core URL: https://github.com/apache/spark/pull/27931#issuecomment-599883659 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core
AmplabJenkins removed a comment on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core URL: https://github.com/apache/spark/pull/27931#issuecomment-599883659 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core
AmplabJenkins removed a comment on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core URL: https://github.com/apache/spark/pull/27931#issuecomment-599883664 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119903/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core
AmplabJenkins commented on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core URL: https://github.com/apache/spark/pull/27931#issuecomment-599883664 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119903/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
AmplabJenkins removed a comment on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp URL: https://github.com/apache/spark/pull/27906#issuecomment-599883374 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24635/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
AmplabJenkins removed a comment on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp URL: https://github.com/apache/spark/pull/27906#issuecomment-599883371 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
AmplabJenkins commented on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp URL: https://github.com/apache/spark/pull/27906#issuecomment-599883371 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
AmplabJenkins commented on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp URL: https://github.com/apache/spark/pull/27906#issuecomment-599883374 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24635/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core
SparkQA removed a comment on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core URL: https://github.com/apache/spark/pull/27931#issuecomment-599843442 **[Test build #119903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119903/testReport)** for PR 27931 at commit [`d554d90`](https://github.com/apache/spark/commit/d554d90ac56da5c3c33550e94a44caefb157cf36). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core
SparkQA commented on issue #27931: [SPARK-31002][CORE][DOC][FOLLOWUP] Add version information to the configuration of Core URL: https://github.com/apache/spark/pull/27931#issuecomment-599883135 **[Test build #119903 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119903/testReport)** for PR 27931 at commit [`d554d90`](https://github.com/apache/spark/commit/d554d90ac56da5c3c33550e94a44caefb157cf36). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
SparkQA commented on issue #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp URL: https://github.com/apache/spark/pull/27906#issuecomment-599883117 **[Test build #119909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119909/testReport)** for PR 27906 at commit [`d1ab415`](https://github.com/apache/spark/commit/d1ab415d523774efa711db7c2173c21fb5945210). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] stczwd commented on a change in pull request #27900: [SPARK-31100][SQL] Check namespace existens when setting namespace
stczwd commented on a change in pull request #27900: [SPARK-31100][SQL] Check
namespace existens when setting namespace
URL: https://github.com/apache/spark/pull/27900#discussion_r393449523
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala
##
@@ -106,13 +106,14 @@ class CatalogManager(
}
def setCurrentNamespace(namespace: Array[String]): Unit = synchronized {
-if (currentCatalog.name() == SESSION_CATALOG_NAME) {
- if (namespace.length != 1) {
+currentCatalog match {
+ case _ if currentCatalog.name() == SESSION_CATALOG_NAME &&
namespace.length == 1 =>
+v1SessionCatalog.setCurrentDatabase(namespace.head)
+ case catalog: SupportsNamespaces if catalog.namespaceExists(namespace) =>
+logInfo(s"set current namespace to ${namespace.mkString(".")}")
+_currentNamespace = Some(namespace)
+ case _ =>
throw new NoSuchNamespaceException(namespace)
Review comment:
> It's very likely that the underlying catalog can't report namespace
existence directly, but can throw namespace not found error when calling APIs
like `loadTable(ident)`
First, the user needs to sense whether the namespace exists.
Secondly, Those who cannot obtain the namespace status can return true
directly in namespaceExists.
Finally, for those don't support namespaces, users should be allowed to set
the defaultNamespace, I will add this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add
SHOW VIEWS command
URL: https://github.com/apache/spark/pull/27897#discussion_r393428635
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
##
@@ -530,6 +523,16 @@ class ResolveSessionCatalog(
replace,
viewType)
+case ShowViewsStatement(SessionCatalogAndNamespace(_, ns), pattern) =>
+ val namespace = if (ns.isEmpty) {
+None
+ } else if (ns.length == 1) {
+Some(ns.head)
+ } else {
+throw new AnalysisException(s"The database name is not valid:
${ns.quoted}")
Review comment:
When creating database/namespace like `CREATE NAMESPACE ns1.ns2`, it throws
invalid database name error. Does it related V2 Session Catalog or some other
reasons? Sorry for my lack of related knowledge. I'm keep investigating this.
Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393448225
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -83,9 +85,9 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
2177456523456789L,
11858049903010203L).foreach { micros =>
DateTimeTestUtils.outstandingZoneIds.foreach { zoneId =>
- val formatter = TimestampFormatter(pattern, zoneId)
- val timestamp = formatter.format(micros)
- val parsed = formatter.parse(timestamp)
+ val timestamp = TimestampFormatter(pattern, zoneId).format(micros)
+ val parsed = TimestampFormatter(
+pattern, zoneId, needVarLengthSecondFraction =
true).parse(timestamp)
Review comment:
this change is needed because it is how the actual roundtrip goes no matter
what the input is
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r39344
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -64,7 +65,8 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
DateTimeTestUtils.outstandingTimezonesIds.foreach { zoneId =>
val formatter = TimestampFormatter(
"-MM-dd'T'HH:mm:ss.SS",
-DateTimeUtils.getZoneId(zoneId))
+DateTimeUtils.getZoneId(zoneId),
+needVarLengthSecondFraction = true)
Review comment:
this one is not needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit
dongjoon-hyun commented on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit URL: https://github.com/apache/spark/pull/27422#issuecomment-599877544 Thank you again for your first contribution, @PedroRossi . You are added to the Apache Spark contributor group and SPARK-25355 is assigned to you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit
dongjoon-hyun closed pull request #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit URL: https://github.com/apache/spark/pull/27422 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit
dongjoon-hyun commented on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit URL: https://github.com/apache/spark/pull/27422#issuecomment-599876655 The failure is irrelevant to this. I verified the relevant tests locally. Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns
AmplabJenkins removed a comment on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns URL: https://github.com/apache/spark/pull/27924#issuecomment-599875777 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119898/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns
AmplabJenkins removed a comment on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns URL: https://github.com/apache/spark/pull/27924#issuecomment-599875773 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns
AmplabJenkins commented on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns URL: https://github.com/apache/spark/pull/27924#issuecomment-599875773 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns
AmplabJenkins commented on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns URL: https://github.com/apache/spark/pull/27924#issuecomment-599875777 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119898/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
yaooqinn commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393442908
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -44,7 +44,8 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
DateTimeTestUtils.outstandingTimezonesIds.foreach { zoneId =>
val formatter = TimestampFormatter(
"-MM-dd'T'HH:mm:ss.SS",
-DateTimeUtils.getZoneId(zoneId))
+DateTimeUtils.getZoneId(zoneId),
+needVarLengthSecondFraction = true)
Review comment:
We are not going to use fixed-length formatter for parsing right? So I
guess test that one is not needed anymore. the var-length formatter should work
fine with fixed inputs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns
SparkQA removed a comment on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns URL: https://github.com/apache/spark/pull/27924#issuecomment-599822109 **[Test build #119898 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119898/testReport)** for PR 27924 at commit [`85d5324`](https://github.com/apache/spark/commit/85d53248685935f317cefdf80a4d66b4ef03809d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns
SparkQA commented on issue #27924: [SPARK-31164][SQL] Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns URL: https://github.com/apache/spark/pull/27924#issuecomment-599875280 **[Test build #119898 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119898/testReport)** for PR 27924 at commit [`85d5324`](https://github.com/apache/spark/commit/85d53248685935f317cefdf80a4d66b4ef03809d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit
AmplabJenkins removed a comment on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit URL: https://github.com/apache/spark/pull/27422#issuecomment-599873687 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119901/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit
AmplabJenkins removed a comment on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit URL: https://github.com/apache/spark/pull/27422#issuecomment-599873684 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit
SparkQA removed a comment on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit URL: https://github.com/apache/spark/pull/27422#issuecomment-599839487 **[Test build #119901 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119901/testReport)** for PR 27422 at commit [`d8d7f7d`](https://github.com/apache/spark/commit/d8d7f7db21c937a0ca470150f39499f1558c648f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit
AmplabJenkins commented on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit URL: https://github.com/apache/spark/pull/27422#issuecomment-599873687 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119901/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit
AmplabJenkins commented on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit URL: https://github.com/apache/spark/pull/27422#issuecomment-599873684 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27895: [SPARK-31138][ML] Add ANOVA Selector for continuous features and categorical labels
zhengruifeng commented on a change in pull request #27895: [SPARK-31138][ML]
Add ANOVA Selector for continuous features and categorical labels
URL: https://github.com/apache/spark/pull/27895#discussion_r393431751
##
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ANOVATest.scala
##
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.stat
+
+import org.apache.commons.math3.distribution.FDistribution
+
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml.feature.LabeledPoint
+import org.apache.spark.ml.linalg.{Vector, Vectors, VectorUDT}
+import org.apache.spark.ml.util.{MetadataUtils, SchemaUtils}
+import org.apache.spark.sql._
+import org.apache.spark.sql.functions._
+import org.apache.spark.util.collection.OpenHashMap
+
+
+/**
+ * ANOVA Test
+ */
+@Since("3.1.0")
+object ANOVATest {
+
+ /** Used to construct output schema of tests */
+ private case class ANOVAResult(
+ pValues: Vector,
+ degreesOfFreedom: Array[Long],
+ fValues: Vector)
+
+ /**
+ * @param dataset DataFrame of categorical labels and continuous features.
+ * @param featuresCol Name of features column in dataset, of type `Vector`
(`VectorUDT`)
+ * @param labelCol Name of label column in dataset, of any numerical type
+ * @return DataFrame containing the test result for every feature against
the label.
+ * This DataFrame will contain a single Row with the following
fields:
+ * - `pValues: Vector`
+ * - `degreesOfFreedom: Array[Long]`
+ * - `fValues: Vector`
+ * Each of these fields has one value per feature.
+ */
+ @Since("3.1.0")
+ def test(dataset: DataFrame, featuresCol: String, labelCol: String):
DataFrame = {
+val spark = dataset.sparkSession
+val testResults = testClassification(dataset, featuresCol, labelCol)
+val pValues: Vector = Vectors.dense(testResults.map(_.pValue))
+val degreesOfFreedom: Array[Long] = testResults.map(_.degreesOfFreedom)
+val fValues: Vector = Vectors.dense(testResults.map(_.statistic))
+spark.createDataFrame(
+ Seq(new ANOVAResult(pValues, degreesOfFreedom, fValues)))
+ }
+
+ /**
+ * @param dataset DataFrame of categorical labels and continuous features.
+ * @param featuresCol Name of features column in dataset, of type `Vector`
(`VectorUDT`)
+ * @param labelCol Name of label column in dataset, of any numerical type
+ * @return Array containing the ANOVATestResult for every feature against the
+ * label.
+ */
+ private[ml] def testClassification(
+ dataset: Dataset[_],
+ featuresCol: String,
+ labelCol: String): Array[SelectionTestResult] = {
+
+val spark = dataset.sparkSession
+import spark.implicits._
+
+SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
+SchemaUtils.checkNumericType(dataset.schema, labelCol)
+
+val labeledPointRdd = dataset.select(col(labelCol).cast("double"),
col(featuresCol))
+ .as[(Double, Vector)]
+ .rdd.map { case (label, features) => LabeledPoint(label, features) }
+
+val numFeatures = MetadataUtils.getNumFeatures(dataset, featuresCol)
+val Row(numSamples: Long, numClasses: Long) =
+ dataset.select(count(labelCol), countDistinct(labelCol)).head
+
+labeledPointRdd.flatMap { case LabeledPoint(label, features) =>
+ features.iterator.map { case (col, value) =>
+(col, (label, value, value * value))
+ }
+}.aggregateByKey[(Double, Double, OpenHashMap[Double, Double],
OpenHashMap[Double, Long])](
+ (0.0, 0.0, new OpenHashMap[Double, Double], new OpenHashMap[Double,
Long]))(
+ seqOp = {
+case (
+ (sum: Double, sumOfSq: Double, mapOfSumPerClass, mapOfCountPerClass),
+ (label, feature, featureSq)
+ ) =>
+ mapOfSumPerClass.changeValue(label, feature, _ + feature)
+ mapOfCountPerClass.changeValue(label, 1L, _ + 1L)
+ (mapOfSumPerClass, mapOfCountPerClass)
+ (sum + feature, sumOfSq + featureSq, mapOfSumPerClass,
mapOfCountPerClass)
+ },
+ combOp = {
+case (
+ (sum1, sumOfSq1, mapOfSumPerClass1,
[GitHub] [spark] SparkQA commented on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit
SparkQA commented on issue #27422: [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit URL: https://github.com/apache/spark/pull/27422#issuecomment-599873451 **[Test build #119901 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119901/testReport)** for PR 27422 at commit [`d8d7f7d`](https://github.com/apache/spark/commit/d8d7f7db21c937a0ca470150f39499f1558c648f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599872529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24634/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599872529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24634/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599872526 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599872526 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27930: [SQL][MINOR] Update the DataFrameWriter.bucketBy comment
cloud-fan commented on a change in pull request #27930: [SQL][MINOR] Update the
DataFrameWriter.bucketBy comment
URL: https://github.com/apache/spark/pull/27930#discussion_r393440110
##
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##
@@ -197,8 +197,8 @@ final class DataFrameWriter[T] private[sql](ds:
Dataset[T]) {
}
/**
- * Buckets the output by the given columns. If specified, the output is laid
out on the file
- * system similar to Hive's bucketing scheme.
+ * Buckets the output by the given columns. Note that the output follows a
Spark SQL specific
+ * bucketing scheme based on the Hive scheme.
Review comment:
I feel the original one is clear enough as it says "similar to". Maybe we
can add one more sentence: `..., but with a different bucket hash function and
is not compatible with Hive's bucketing.`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
SparkQA commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599872286 **[Test build #119908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119908/testReport)** for PR 27932 at commit [`78f8021`](https://github.com/apache/spark/commit/78f802126f7f935716d7e1783cbd891199e8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599871095 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119906/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599871089 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
SparkQA removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599869183 **[Test build #119906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119906/testReport)** for PR 27932 at commit [`1bf9e7d`](https://github.com/apache/spark/commit/1bf9e7dd0497c788ec05a5d3a8d757ef30c3131e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599871089 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599871095 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119906/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
SparkQA commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599871074 **[Test build #119906 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119906/testReport)** for PR 27932 at commit [`1bf9e7d`](https://github.com/apache/spark/commit/1bf9e7dd0497c788ec05a5d3a8d757ef30c3131e). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27900: [SPARK-31100][SQL] Check namespace existens when setting namespace
cloud-fan commented on a change in pull request #27900: [SPARK-31100][SQL]
Check namespace existens when setting namespace
URL: https://github.com/apache/spark/pull/27900#discussion_r393438037
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala
##
@@ -106,13 +106,14 @@ class CatalogManager(
}
def setCurrentNamespace(namespace: Array[String]): Unit = synchronized {
-if (currentCatalog.name() == SESSION_CATALOG_NAME) {
- if (namespace.length != 1) {
+currentCatalog match {
+ case _ if currentCatalog.name() == SESSION_CATALOG_NAME &&
namespace.length == 1 =>
+v1SessionCatalog.setCurrentDatabase(namespace.head)
+ case catalog: SupportsNamespaces if catalog.namespaceExists(namespace) =>
+logInfo(s"set current namespace to ${namespace.mkString(".")}")
+_currentNamespace = Some(namespace)
+ case _ =>
throw new NoSuchNamespaceException(namespace)
Review comment:
And we should allow users to set current namespace when working with these
catalogs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27900: [SPARK-31100][SQL] Check namespace existens when setting namespace
cloud-fan commented on a change in pull request #27900: [SPARK-31100][SQL]
Check namespace existens when setting namespace
URL: https://github.com/apache/spark/pull/27900#discussion_r393437946
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala
##
@@ -106,13 +106,14 @@ class CatalogManager(
}
def setCurrentNamespace(namespace: Array[String]): Unit = synchronized {
-if (currentCatalog.name() == SESSION_CATALOG_NAME) {
- if (namespace.length != 1) {
+currentCatalog match {
+ case _ if currentCatalog.name() == SESSION_CATALOG_NAME &&
namespace.length == 1 =>
+v1SessionCatalog.setCurrentDatabase(namespace.head)
+ case catalog: SupportsNamespaces if catalog.namespaceExists(namespace) =>
+logInfo(s"set current namespace to ${namespace.mkString(".")}")
+_currentNamespace = Some(namespace)
+ case _ =>
throw new NoSuchNamespaceException(namespace)
Review comment:
It's very likely that the underlying catalog can't report namespace
existence directly, but can throw namespace not found error when calling APIs
like `loadTable(ident)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393437428
##
File path: sql/core/src/test/resources/sql-tests/results/datetime.sql.out
##
@@ -391,3 +391,163 @@ select date '2001-10-01' - date '2001-09-28'
struct
-- !query output
3 days
+
+
+-- !query
+select to_timestamp('2019-10-06 10:11:12.', '-MM-dd HH:mm:ss.SS[zzz]')
+-- !query schema
+struct
+-- !query output
+NULL
+
+
+-- !query
+select to_timestamp('2019-10-06 10:11:12.0', '-MM-dd HH:mm:ss.SS[zzz]')
+-- !query schema
+struct
+-- !query output
+2019-10-06 10:11:12
+
+
+-- !query
+select to_timestamp('2019-10-06 10:11:12.1', '-MM-dd HH:mm:ss.SS[zzz]')
+-- !query schema
+struct
+-- !query output
+2019-10-06 10:11:12.1
+
+
+-- !query
+select to_timestamp('2019-10-06 10:11:12.12', '-MM-dd
HH:mm:ss.SS[zzz]')
+-- !query schema
+struct
+-- !query output
+2019-10-06 10:11:12.12
+
+
+-- !query
+select to_timestamp('2019-10-06 10:11:12.123UTC', '-MM-dd
HH:mm:ss.SS[zzz]')
+-- !query schema
+struct
+-- !query output
+2019-10-06 03:11:12.123
+
+
+-- !query
+select to_timestamp('2019-10-06 10:11:12.1234', '-MM-dd
HH:mm:ss.SS[zzz]')
+-- !query schema
+struct
+-- !query output
+2019-10-06 10:11:12.1234
+
+
+-- !query
+select to_timestamp('2019-10-06 10:11:12.12345CST', '-MM-dd
HH:mm:ss.SS[zzz]')
+-- !query schema
+struct
+-- !query output
+2019-10-06 08:11:12.12345
+
+
+-- !query
+select to_timestamp('2019-10-06 10:11:12.123456PST', '-MM-dd
HH:mm:ss.SS[zzz]')
+-- !query schema
+struct
+-- !query output
+2019-10-06 10:11:12.123456
+
+
+-- !query
+select to_timestamp('2019-10-06 10:11:12.1234567PST', '-MM-dd
HH:mm:ss.SS[zzz]')
+-- !query schema
+struct
+-- !query output
+NULL
+
+
+-- !query
+select to_timestamp('123456 2019-10-06 10:11:12.123456PST', 'SS -MM-dd
HH:mm:ss.SS[zzz]')
+-- !query schema
+struct
+-- !query output
+2019-10-06 10:11:12.123456
+
+
+-- !query
+select to_timestamp('223456 2019-10-06 10:11:12.123456PST', 'SS -MM-dd
HH:mm:ss.SS[zzz]')
+-- !query schema
+struct
+-- !query output
+NULL
+
+
+-- !query
+select to_timestamp('2019-10-06 10:11:12.1234', '-MM-dd HH:mm:ss.[SS]')
+-- !query schema
+struct
+-- !query output
+2019-10-06 10:11:12.1234
+
+
+-- !query
+select to_timestamp('2019-10-06 10:11:12.123', '-MM-dd HH:mm:ss[.SS]')
Review comment:
can we test the optional case? `to_timestamp('2019-10-06 10:11:12',
'-MM-dd HH:mm:ss[.SS]')`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add
SHOW VIEWS command
URL: https://github.com/apache/spark/pull/27897#discussion_r393428635
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
##
@@ -530,6 +523,16 @@ class ResolveSessionCatalog(
replace,
viewType)
+case ShowViewsStatement(SessionCatalogAndNamespace(_, ns), pattern) =>
+ val namespace = if (ns.isEmpty) {
+None
+ } else if (ns.length == 1) {
+Some(ns.head)
+ } else {
+throw new AnalysisException(s"The database name is not valid:
${ns.quoted}")
Review comment:
When creating database/namespace like `CREATE NAMESPACE ns1.ns2`, it throws
invalid database name error. Does it related V2 Session Catalog or some other
reasons? Sorry for my lack of related knowledge. I'm keep investigating this.
Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599869433 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
AmplabJenkins removed a comment on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1 URL: https://github.com/apache/spark/pull/27860#issuecomment-599869435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24633/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins removed a comment on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599869437 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24632/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393436770
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -83,9 +85,9 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
2177456523456789L,
11858049903010203L).foreach { micros =>
DateTimeTestUtils.outstandingZoneIds.foreach { zoneId =>
- val formatter = TimestampFormatter(pattern, zoneId)
- val timestamp = formatter.format(micros)
- val parsed = formatter.parse(timestamp)
+ val timestamp = TimestampFormatter(pattern, zoneId).format(micros)
+ val parsed = TimestampFormatter(
+pattern, zoneId, needVarLengthSecondFraction =
true).parse(timestamp)
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
AmplabJenkins removed a comment on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1 URL: https://github.com/apache/spark/pull/27860#issuecomment-599869428 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393436722
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -64,7 +65,8 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
DateTimeTestUtils.outstandingTimezonesIds.foreach { zoneId =>
val formatter = TimestampFormatter(
"-MM-dd'T'HH:mm:ss.SS",
-DateTimeUtils.getZoneId(zoneId))
+DateTimeUtils.getZoneId(zoneId),
+needVarLengthSecondFraction = true)
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
AmplabJenkins commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1 URL: https://github.com/apache/spark/pull/27860#issuecomment-599869435 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24633/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL] Parsing seconds fraction with variable length for timestamp
cloud-fan commented on a change in pull request #27906: [SPARK-31150][SQL]
Parsing seconds fraction with variable length for timestamp
URL: https://github.com/apache/spark/pull/27906#discussion_r393436691
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
##
@@ -44,7 +44,8 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
DateTimeTestUtils.outstandingTimezonesIds.foreach { zoneId =>
val formatter = TimestampFormatter(
"-MM-dd'T'HH:mm:ss.SS",
-DateTimeUtils.getZoneId(zoneId))
+DateTimeUtils.getZoneId(zoneId),
+needVarLengthSecondFraction = true)
Review comment:
is this really needed? The test input is fixed: `2018-12-02T10:11:12.001234`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
AmplabJenkins commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1 URL: https://github.com/apache/spark/pull/27860#issuecomment-599869428 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599869437 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24632/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
AmplabJenkins commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599869433 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16
SparkQA commented on issue #27932: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27932#issuecomment-599869183 **[Test build #119906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119906/testReport)** for PR 27932 at commit [`1bf9e7d`](https://github.com/apache/spark/commit/1bf9e7dd0497c788ec05a5d3a8d757ef30c3131e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
AmplabJenkins removed a comment on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1 URL: https://github.com/apache/spark/pull/27860#issuecomment-599868872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119905/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
SparkQA commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1 URL: https://github.com/apache/spark/pull/27860#issuecomment-599869179 **[Test build #119907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119907/testReport)** for PR 27860 at commit [`51d090d`](https://github.com/apache/spark/commit/51d090dbcd2fea7eb7e8d4a852e53e89c017284e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
HeartSaVioR commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1 URL: https://github.com/apache/spark/pull/27860#issuecomment-599869036 retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
SparkQA removed a comment on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1 URL: https://github.com/apache/spark/pull/27860#issuecomment-599867658 **[Test build #119905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119905/testReport)** for PR 27860 at commit [`51d090d`](https://github.com/apache/spark/commit/51d090dbcd2fea7eb7e8d4a852e53e89c017284e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
SparkQA commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1 URL: https://github.com/apache/spark/pull/27860#issuecomment-599868855 **[Test build #119905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119905/testReport)** for PR 27860 at commit [`51d090d`](https://github.com/apache/spark/commit/51d090dbcd2fea7eb7e8d4a852e53e89c017284e). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
AmplabJenkins commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1 URL: https://github.com/apache/spark/pull/27860#issuecomment-599868864 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1
AmplabJenkins commented on issue #27860: [WIP][SPARK-31101][SQL][BUILD] Upgrade Janino to 3.1.1 URL: https://github.com/apache/spark/pull/27860#issuecomment-599868872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119905/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
