[GitHub] AmplabJenkins commented on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits
AmplabJenkins commented on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits URL: https://github.com/apache/spark/pull/23266#issuecomment-446880898 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6067/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
SparkQA commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446880801 **[Test build #100079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100079/testReport)** for PR 23293 at commit [`fad25e2`](https://github.com/apache/spark/commit/fad25e2905271f6971c3147fb289331b9ae7c218). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits
SparkQA commented on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits URL: https://github.com/apache/spark/pull/23266#issuecomment-446880799 **[Test build #100080 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100080/testReport)** for PR 23266 at commit [`99e9fe9`](https://github.com/apache/spark/commit/99e9fe9b0cbc7e62b418e85fdf939f86cf25c8b6). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] MaxGekk commented on a change in pull request #23196: [SPARK-26243][SQL] Use java.time API for parsing timestamps and dates from JSON
MaxGekk commented on a change in pull request #23196: [SPARK-26243][SQL] Use
java.time API for parsing timestamps and dates from JSON
URL: https://github.com/apache/spark/pull/23196#discussion_r241314965
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatter.scala
##
@@ -33,26 +33,37 @@ sealed trait DateTimeFormatter {
def format(us: Long): String
}
+trait FormatterUtils {
+ def zoneId: ZoneId
+ def buildFormatter(pattern: String, locale: Locale):
java.time.format.DateTimeFormatter = {
+new DateTimeFormatterBuilder()
+ .appendPattern(pattern)
+ .parseDefaulting(ChronoField.YEAR_OF_ERA, 1970)
+ .parseDefaulting(ChronoField.MONTH_OF_YEAR, 1)
+ .parseDefaulting(ChronoField.DAY_OF_MONTH, 1)
+ .parseDefaulting(ChronoField.HOUR_OF_DAY, 0)
+ .parseDefaulting(ChronoField.MINUTE_OF_HOUR, 0)
+ .parseDefaulting(ChronoField.SECOND_OF_MINUTE, 0)
Review comment:
Let's say ff you need to parse `Dec 2018`, the parser doesn't know what day
it should use by default.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
SparkQA removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446880801 **[Test build #100079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100079/testReport)** for PR 23293 at commit [`fad25e2`](https://github.com/apache/spark/commit/fad25e2905271f6971c3147fb289331b9ae7c218). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446891546 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100079/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] mgaido91 commented on a change in pull request #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subqueries
mgaido91 commented on a change in pull request #23057: [SPARK-26078][SQL] Dedup
self-join attributes on IN subqueries
URL: https://github.com/apache/spark/pull/23057#discussion_r241322725
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##
@@ -54,10 +54,7 @@ object RewritePredicateSubquery extends Rule[LogicalPlan]
with PredicateHelper {
val aliasMap = AttributeMap(duplicates.map { dup =>
dup -> Alias(dup, dup.toString)()
}.toSeq)
-val aliasedExpressions = right.output.map { ref =>
- aliasMap.getOrElse(ref, ref)
-}
-val newRight = Project(aliasedExpressions, right)
+val newRight = rewriteDedupPlan(right, aliasMap)
val newJoinCond = joinCond.map { condExpr =>
Review comment:
yes, I actually think this is useless. Let me try and remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446876945 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100076/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
SparkQA removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446865991 **[Test build #100073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100073/testReport)** for PR 23068 at commit [`0a63604`](https://github.com/apache/spark/commit/0a636049ecc721cdd31cd676fce79aeb6582dd7c). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories
AmplabJenkins removed a comment on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories URL: https://github.com/apache/spark/pull/23108#issuecomment-446877144 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule
AmplabJenkins removed a comment on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446876990 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
AmplabJenkins removed a comment on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-446876979 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule
AmplabJenkins removed a comment on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446877000 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100075/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max&
AmplabJenkins removed a comment on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max& URL: https://github.com/apache/spark/pull/23304#issuecomment-446877125 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100071/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories
AmplabJenkins removed a comment on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories URL: https://github.com/apache/spark/pull/23108#issuecomment-446877148 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100072/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] gengliangwang commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
gengliangwang commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446877693 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
AmplabJenkins removed a comment on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-446876985 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100074/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446876988 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100073/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446879560 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6065/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446879554 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446879554 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on a change in pull request #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits
HyukjinKwon commented on a change in pull request #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits URL: https://github.com/apache/spark/pull/23266#discussion_r241304251 ## File path: sql/core/src/main/java/org/apache/spark/sql/sources/v2/Table.java ## @@ -43,17 +41,8 @@ String name(); /** - * Returns the schema of this table. + * Returns the schema of this table. If the table is not readable and doesn't have a schema, an + * empty schema can be returned here. */ StructType schema(); Review comment: Ahh, I regret that I missed today's meeting. Sounds good to me and that exactly addresses my concern. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23307: [SPARK-26335][SQL] Add a property for Dataset#show not to care about wide characters when padding them
HyukjinKwon commented on issue #23307: [SPARK-26335][SQL] Add a property for Dataset#show not to care about wide characters when padding them URL: https://github.com/apache/spark/pull/23307#issuecomment-446891328 adding @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] MaxGekk commented on a change in pull request #23196: [SPARK-26243][SQL] Use java.time API for parsing timestamps and dates from JSON
MaxGekk commented on a change in pull request #23196: [SPARK-26243][SQL] Use
java.time API for parsing timestamps and dates from JSON
URL: https://github.com/apache/spark/pull/23196#discussion_r241316473
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatter.scala
##
@@ -116,29 +127,32 @@ sealed trait DateFormatter {
class Iso8601DateFormatter(
pattern: String,
-timeZone: TimeZone,
-locale: Locale) extends DateFormatter {
+locale: Locale) extends DateFormatter with FormatterUtils {
- val dateTimeFormatter = new Iso8601DateTimeFormatter(pattern, timeZone,
locale)
+ val zoneId = ZoneId.of("GMT")
Review comment:
Thanks for the link. I probably read the same when I thought what is better
to use. What confused me is **GMT is a time zone** vs **UTC is not a time zone**
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446891542 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
SparkQA commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446891456 **[Test build #100079 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100079/testReport)** for PR 23293 at commit [`fad25e2`](https://github.com/apache/spark/commit/fad25e2905271f6971c3147fb289331b9ae7c218). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax
SparkQA commented on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax URL: https://github.com/apache/spark/pull/23295#issuecomment-446892694 **[Test build #100081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100081/testReport)** for PR 23295 at commit [`361ed5c`](https://github.com/apache/spark/commit/361ed5c51f113a0cea6ec2d6d7dcfd9c877a4694). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
SparkQA commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446877979 **[Test build #100078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100078/testReport)** for PR 23068 at commit [`0a63604`](https://github.com/apache/spark/commit/0a636049ecc721cdd31cd676fce79aeb6582dd7c). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon edited a comment on issue #23307: [SPARK-26335][SQL] Add a property for Dataset#show not to care about wide characters when padding them
HyukjinKwon edited a comment on issue #23307: [SPARK-26335][SQL] Add a property
for Dataset#show not to care about wide characters when padding them
URL: https://github.com/apache/spark/pull/23307#issuecomment-446890583
I think it makes more sense that OASIS side implements one method, for
instance, `showAsTable`
```scala
implicit class OASIS(df: DataFrame) {
def showAsTable() = {
// Mimic `Dataset.showString`
}
}
```
and
```scala
dataset.showAsTable
```
I think `df.show()` does not have a guarantee about how it's printed in
console, and the purpose of this API is just to show a pretty string rather
than parsing and using it. This configuration looks too specific to OASIS.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] mgaido91 commented on issue #23285: [SPARK-26224][SQL] Avoid creating many project on subsequent calls to withColumn
mgaido91 commented on issue #23285: [SPARK-26224][SQL] Avoid creating many project on subsequent calls to withColumn URL: https://github.com/apache/spark/pull/23285#issuecomment-446893251 @HeartSaVioR I'd rather say that `withColumns` can be easily done as well using select. So I am not sure it is so useful to expose it. There is already a "workaround" for this problem, this PR just meant to avoid that issue since many people use it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule
SparkQA commented on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446904518 **[Test build #100082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100082/testReport)** for PR 23273 at commit [`1cca847`](https://github.com/apache/spark/commit/1cca8475e999139ab8f22f5cb583d9edce3cf550). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on a change in pull request #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
cloud-fan commented on a change in pull request #23294:
[SPARK-26265][Core][Followup] Put freePage into a finally block
URL: https://github.com/apache/spark/pull/23294#discussion_r241300985
##
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##
@@ -262,36 +262,39 @@ private void advanceToNextPage() {
// reference to the page to free and free it after releasing the lock of
`MapIterator`.
MemoryBlock pageToFree = null;
- synchronized (this) {
-int nextIdx = dataPages.indexOf(currentPage) + 1;
-if (destructive && currentPage != null) {
- dataPages.remove(currentPage);
- pageToFree = currentPage;
- nextIdx --;
-}
-if (dataPages.size() > nextIdx) {
- currentPage = dataPages.get(nextIdx);
- pageBaseObject = currentPage.getBaseObject();
- offsetInPage = currentPage.getBaseOffset();
- recordsInPage = UnsafeAlignedOffset.getSize(pageBaseObject,
offsetInPage);
- offsetInPage += UnsafeAlignedOffset.getUaoSize();
-} else {
- currentPage = null;
- if (reader != null) {
-handleFailedDelete();
+ try {
+synchronized (this) {
+ int nextIdx = dataPages.indexOf(currentPage) + 1;
+ if (destructive && currentPage != null) {
+dataPages.remove(currentPage);
+pageToFree = currentPage;
+nextIdx--;
}
- try {
-Closeables.close(reader, /* swallowIOException = */ false);
Review comment:
shall we close the reader outside of the `synchronized` block? Then we don't
need to extra `try catch`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule
SparkQA commented on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446876933 **[Test build #100075 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100075/testReport)** for PR 23273 at commit [`1cca847`](https://github.com/apache/spark/commit/1cca8475e999139ab8f22f5cb583d9edce3cf550). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446876982 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max&
SparkQA commented on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max& URL: https://github.com/apache/spark/pull/23304#issuecomment-446876937 **[Test build #100071 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100071/testReport)** for PR 23304 at commit [`defee03`](https://github.com/apache/spark/commit/defee03947e3db885a34af546716b36cf62249f6). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class TypedMaxDouble[IN](val f: IN => Double) extends Aggregator[IN, Double, Double] ` * `class TypedMaxLong[IN](val f: IN => Long) extends Aggregator[IN, Long, Long] ` * `class TypedMinDouble[IN](val f: IN => Double) extends Aggregator[IN, Double, Double] ` * `class TypedMinLong[IN](val f: IN => Long) extends Aggregator[IN, Long, Long] ` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446876945 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100076/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories
AmplabJenkins commented on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories URL: https://github.com/apache/spark/pull/23108#issuecomment-446877148 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100072/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
SparkQA removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446875353 **[Test build #100076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100076/testReport)** for PR 23293 at commit [`fad25e2`](https://github.com/apache/spark/commit/fad25e2905271f6971c3147fb289331b9ae7c218). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
SparkQA removed a comment on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-446867109 **[Test build #100074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100074/testReport)** for PR 23294 at commit [`630281b`](https://github.com/apache/spark/commit/630281b59503b88ed5ba14e1b8e5029e1589c478). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule
AmplabJenkins commented on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446877000 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100075/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any
AmplabJenkins removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-446876944 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446876988 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100073/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule
SparkQA removed a comment on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446874129 **[Test build #100075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100075/testReport)** for PR 23273 at commit [`1cca847`](https://github.com/apache/spark/commit/1cca8475e999139ab8f22f5cb583d9edce3cf550). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max&
AmplabJenkins commented on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max& URL: https://github.com/apache/spark/pull/23304#issuecomment-446877125 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100071/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any
SparkQA commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-446876931 **[Test build #100077 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100077/testReport)** for PR 23275 at commit [`cc08bf1`](https://github.com/apache/spark/commit/cc08bf1590417e706e81eb93a779dc0cf63cd7d6). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
AmplabJenkins commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-446876985 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100074/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
SparkQA commented on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446876924 **[Test build #100073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100073/testReport)** for PR 23068 at commit [`0a63604`](https://github.com/apache/spark/commit/0a636049ecc721cdd31cd676fce79aeb6582dd7c). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any
AmplabJenkins commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-446876956 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100077/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule
AmplabJenkins commented on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446876990 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
SparkQA commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-446876929 **[Test build #100074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100074/testReport)** for PR 23294 at commit [`630281b`](https://github.com/apache/spark/commit/630281b59503b88ed5ba14e1b8e5029e1589c478). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446876941 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any
SparkQA removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-446875362 **[Test build #100077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100077/testReport)** for PR 23275 at commit [`cc08bf1`](https://github.com/apache/spark/commit/cc08bf1590417e706e81eb93a779dc0cf63cd7d6). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any
AmplabJenkins commented on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-446876944 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
AmplabJenkins commented on issue #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block URL: https://github.com/apache/spark/pull/23294#issuecomment-446876979 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446876941 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories
SparkQA commented on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories URL: https://github.com/apache/spark/pull/23108#issuecomment-446876936 **[Test build #100072 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100072/testReport)** for PR 23108 at commit [`fef8c68`](https://github.com/apache/spark/commit/fef8c6845ccce792474a8c65ee6ebfd9624bb4b4). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max&
AmplabJenkins removed a comment on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max& URL: https://github.com/apache/spark/pull/23304#issuecomment-446877120 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any
AmplabJenkins removed a comment on issue #23275: [SPARK-26323][SQL] Scala UDF should still check input types even if some inputs are of type Any URL: https://github.com/apache/spark/pull/23275#issuecomment-446876956 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100077/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max&
SparkQA removed a comment on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max& URL: https://github.com/apache/spark/pull/23304#issuecomment-446853689 **[Test build #100071 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100071/testReport)** for PR 23304 at commit [`defee03`](https://github.com/apache/spark/commit/defee03947e3db885a34af546716b36cf62249f6). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
SparkQA commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446876926 **[Test build #100076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100076/testReport)** for PR 23293 at commit [`fad25e2`](https://github.com/apache/spark/commit/fad25e2905271f6971c3147fb289331b9ae7c218). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max&
AmplabJenkins commented on issue #23304: [SPARK-26353][SQL]Add typed aggregate functions: max& URL: https://github.com/apache/spark/pull/23304#issuecomment-446877120 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories
AmplabJenkins commented on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories URL: https://github.com/apache/spark/pull/23108#issuecomment-446877144 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446876982 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories
SparkQA removed a comment on issue #23108: [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories URL: https://github.com/apache/spark/pull/23108#issuecomment-446855600 **[Test build #100072 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100072/testReport)** for PR 23108 at commit [`fef8c68`](https://github.com/apache/spark/commit/fef8c6845ccce792474a8c65ee6ebfd9624bb4b4). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits
HyukjinKwon commented on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits URL: https://github.com/apache/spark/pull/23266#issuecomment-446879815 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
HyukjinKwon commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446880095 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page
AmplabJenkins removed a comment on issue #23068: [SPARK-26098][WebUI] Show associated SQL query in Job page URL: https://github.com/apache/spark/pull/23068#issuecomment-446879560 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6065/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23307: [SPARK-26335][SQL] Add a property for Dataset#show not to care about wide characters when padding them
HyukjinKwon commented on issue #23307: [SPARK-26335][SQL] Add a property for
Dataset#show not to care about wide characters when padding them
URL: https://github.com/apache/spark/pull/23307#issuecomment-446890583
I think it makes more sense that OASIS side implements one method, for
instance, `showAsTable`
```scala
implicit class OASIS(df: DataFrame) {
def showAsTable(): DataFrame = {
// Mimic `Dataset.showString`
}
}
```
and
```scala
dataset.showAsTable
```
I think `df.show()` does not have a guarantee about how it's printed in
console, and the purpose of this API is just to show a pretty string rather
than parsing and using it. This configuration looks too specific to OASIS.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] MaxGekk commented on a change in pull request #23196: [SPARK-26243][SQL] Use java.time API for parsing timestamps and dates from JSON
MaxGekk commented on a change in pull request #23196: [SPARK-26243][SQL] Use
java.time API for parsing timestamps and dates from JSON
URL: https://github.com/apache/spark/pull/23196#discussion_r241315581
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatter.scala
##
@@ -33,26 +33,37 @@ sealed trait DateTimeFormatter {
def format(us: Long): String
}
+trait FormatterUtils {
+ def zoneId: ZoneId
+ def buildFormatter(pattern: String, locale: Locale):
java.time.format.DateTimeFormatter = {
+new DateTimeFormatterBuilder()
+ .appendPattern(pattern)
+ .parseDefaulting(ChronoField.YEAR_OF_ERA, 1970)
+ .parseDefaulting(ChronoField.MONTH_OF_YEAR, 1)
+ .parseDefaulting(ChronoField.DAY_OF_MONTH, 1)
+ .parseDefaulting(ChronoField.HOUR_OF_DAY, 0)
+ .parseDefaulting(ChronoField.MINUTE_OF_HOUR, 0)
+ .parseDefaulting(ChronoField.SECOND_OF_MINUTE, 0)
+ .toFormatter(locale)
+ }
+ def toInstant(temporalAccessor: TemporalAccessor): java.time.Instant = {
+val localDateTime = LocalDateTime.from(temporalAccessor)
+val zonedDateTime = ZonedDateTime.of(localDateTime, zoneId)
+Instant.from(zonedDateTime)
+ }
+}
+
class Iso8601DateTimeFormatter(
pattern: String,
timeZone: TimeZone,
-locale: Locale) extends DateTimeFormatter {
- val formatter = new DateTimeFormatterBuilder()
-.appendPattern(pattern)
-.parseDefaulting(ChronoField.YEAR_OF_ERA, 1970)
-.parseDefaulting(ChronoField.MONTH_OF_YEAR, 1)
-.parseDefaulting(ChronoField.DAY_OF_MONTH, 1)
-.parseDefaulting(ChronoField.HOUR_OF_DAY, 0)
-.parseDefaulting(ChronoField.MINUTE_OF_HOUR, 0)
-.parseDefaulting(ChronoField.SECOND_OF_MINUTE, 0)
-.toFormatter(locale)
+locale: Locale) extends DateTimeFormatter with FormatterUtils {
+ val zoneId = timeZone.toZoneId
+ val formatter = buildFormatter(pattern, locale)
def toInstant(s: String): Instant = {
val temporalAccessor = formatter.parse(s)
if (temporalAccessor.query(TemporalQueries.offset()) == null) {
Review comment:
zone offset is unknown after parsing. For example, if you parse `13-12-2018
09:55:00`, it is unclear in which timezone it is.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] seancxmao commented on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule
seancxmao commented on issue #23273: [MINOR][DOC] Fix comments of ConvertToLocalRelation rule URL: https://github.com/apache/spark/pull/23273#issuecomment-446903185 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446880993 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on a change in pull request #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subqueries
cloud-fan commented on a change in pull request #23057: [SPARK-26078][SQL]
Dedup self-join attributes on IN subqueries
URL: https://github.com/apache/spark/pull/23057#discussion_r241305806
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##
@@ -54,10 +54,7 @@ object RewritePredicateSubquery extends Rule[LogicalPlan]
with PredicateHelper {
val aliasMap = AttributeMap(duplicates.map { dup =>
dup -> Alias(dup, dup.toString)()
}.toSeq)
-val aliasedExpressions = right.output.map { ref =>
- aliasMap.getOrElse(ref, ref)
-}
-val newRight = Project(aliasedExpressions, right)
+val newRight = rewriteDedupPlan(right, aliasMap)
val newJoinCond = joinCond.map { condExpr =>
Review comment:
not related to your PR, but is this corrected? The duplicated attributes in
join condition may refer to left or right child, how can we blindly replace
them with new attributes from right side?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits
AmplabJenkins removed a comment on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits URL: https://github.com/apache/spark/pull/23266#issuecomment-446880898 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6067/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446880993 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446880995 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6066/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits
AmplabJenkins removed a comment on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits URL: https://github.com/apache/spark/pull/23266#issuecomment-446880888 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446880995 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6066/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits
AmplabJenkins commented on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits URL: https://github.com/apache/spark/pull/23266#issuecomment-446880888 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] MaxGekk commented on a change in pull request #23196: [SPARK-26243][SQL] Use java.time API for parsing timestamps and dates from JSON
MaxGekk commented on a change in pull request #23196: [SPARK-26243][SQL] Use
java.time API for parsing timestamps and dates from JSON
URL: https://github.com/apache/spark/pull/23196#discussion_r241317291
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatter.scala
##
@@ -33,26 +33,37 @@ sealed trait DateTimeFormatter {
def format(us: Long): String
}
+trait FormatterUtils {
+ def zoneId: ZoneId
+ def buildFormatter(pattern: String, locale: Locale):
java.time.format.DateTimeFormatter = {
+new DateTimeFormatterBuilder()
+ .appendPattern(pattern)
+ .parseDefaulting(ChronoField.YEAR_OF_ERA, 1970)
+ .parseDefaulting(ChronoField.MONTH_OF_YEAR, 1)
+ .parseDefaulting(ChronoField.DAY_OF_MONTH, 1)
+ .parseDefaulting(ChronoField.HOUR_OF_DAY, 0)
+ .parseDefaulting(ChronoField.MINUTE_OF_HOUR, 0)
+ .parseDefaulting(ChronoField.SECOND_OF_MINUTE, 0)
+ .toFormatter(locale)
+ }
+ def toInstant(temporalAccessor: TemporalAccessor): java.time.Instant = {
Review comment:
Strictly speaking it uses non-default time zone, what about
`toInstantWithZoneId`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] 10110346 commented on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax
10110346 commented on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax URL: https://github.com/apache/spark/pull/23295#issuecomment-446892291 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax
AmplabJenkins removed a comment on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax URL: https://github.com/apache/spark/pull/23295#issuecomment-446894584 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6068/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax
AmplabJenkins commented on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax URL: https://github.com/apache/spark/pull/23295#issuecomment-446894579 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax
AmplabJenkins commented on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax URL: https://github.com/apache/spark/pull/23295#issuecomment-446894584 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6068/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax
AmplabJenkins removed a comment on issue #23295: [MINOR][SQL]Change `ThreadLocal.withInitial` to Scala lambda syntax URL: https://github.com/apache/spark/pull/23295#issuecomment-446894579 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] viirya commented on a change in pull request #23294: [SPARK-26265][Core][Followup] Put freePage into a finally block
viirya commented on a change in pull request #23294:
[SPARK-26265][Core][Followup] Put freePage into a finally block
URL: https://github.com/apache/spark/pull/23294#discussion_r241311302
##
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##
@@ -262,36 +262,39 @@ private void advanceToNextPage() {
// reference to the page to free and free it after releasing the lock of
`MapIterator`.
MemoryBlock pageToFree = null;
- synchronized (this) {
-int nextIdx = dataPages.indexOf(currentPage) + 1;
-if (destructive && currentPage != null) {
- dataPages.remove(currentPage);
- pageToFree = currentPage;
- nextIdx --;
-}
-if (dataPages.size() > nextIdx) {
- currentPage = dataPages.get(nextIdx);
- pageBaseObject = currentPage.getBaseObject();
- offsetInPage = currentPage.getBaseOffset();
- recordsInPage = UnsafeAlignedOffset.getSize(pageBaseObject,
offsetInPage);
- offsetInPage += UnsafeAlignedOffset.getUaoSize();
-} else {
- currentPage = null;
- if (reader != null) {
-handleFailedDelete();
+ try {
+synchronized (this) {
+ int nextIdx = dataPages.indexOf(currentPage) + 1;
+ if (destructive && currentPage != null) {
+dataPages.remove(currentPage);
+pageToFree = currentPage;
+nextIdx--;
}
- try {
-Closeables.close(reader, /* swallowIOException = */ false);
Review comment:
In the second `try`, it accesses `spillWriters`:
```
reader = spillWriters.getFirst().getReader(serializerManager)
```
The `synchronized` `spill` will change `spillWriters`. Seems to be unsafe if
we close the reader and update `reader` like above outside `synchronized`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] MaxGekk commented on a change in pull request #23196: [SPARK-26243][SQL] Use java.time API for parsing timestamps and dates from JSON
MaxGekk commented on a change in pull request #23196: [SPARK-26243][SQL] Use java.time API for parsing timestamps and dates from JSON URL: https://github.com/apache/spark/pull/23196#discussion_r241314004 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ## @@ -77,6 +77,12 @@ private[sql] class JacksonGenerator( private val lineSeparator: String = options.lineSeparatorInWrite + private val timeFormatter = DateTimeFormatter( Review comment: `Date`/`TimestampFormatter` ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446891546 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100079/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
AmplabJenkins removed a comment on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446891542 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subqueries
SparkQA commented on issue #23057: [SPARK-26078][SQL] Dedup self-join attributes on IN subqueries URL: https://github.com/apache/spark/pull/23057#issuecomment-446907889 **[Test build #100083 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100083/testReport)** for PR 23057 at commit [`6528582`](https://github.com/apache/spark/commit/652858213e941f865b61c3d0c542d0db66ba4a57). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] mgaido91 commented on issue #23171: [SPARK-26205][SQL] Optimize In for bytes, shorts, ints
mgaido91 commented on issue #23171: [SPARK-26205][SQL] Optimize In for bytes, shorts, ints URL: https://github.com/apache/spark/pull/23171#issuecomment-446916902 thanks @aokolnychyi. I just have a couple of comments on this: 1 - As @rxin mentioned, now we have InSet for handling Literals and In for handling arbitrary expressions. Since this method works only with literals, I'd rather see it as an alternative execution for InSet rather than for In. Then we might want to convert always (without a threshold) a In containing literals in InSet and let InSet pick the best implementation (either switch or the real InSet). What do you think about this? 2 - I think we may also support longs. We just need to split a long in 2 integers, so with 2 nested switches it would be doable I think. I see this will add complexity and we need to write a dedicated implementation for it, but we may consider this as a followup work. Do you agree on this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on a change in pull request #23196: [SPARK-26243][SQL] Use java.time API for parsing timestamps and dates from JSON
cloud-fan commented on a change in pull request #23196: [SPARK-26243][SQL] Use java.time API for parsing timestamps and dates from JSON URL: https://github.com/apache/spark/pull/23196#discussion_r241355564 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ## @@ -77,6 +77,12 @@ private[sql] class JacksonGenerator( private val lineSeparator: String = options.lineSeparatorInWrite + private val timeFormatter = DateTimeFormatter( Review comment: SGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] cloud-fan commented on a change in pull request #23196: [SPARK-26243][SQL] Use java.time API for parsing timestamps and dates from JSON
cloud-fan commented on a change in pull request #23196: [SPARK-26243][SQL] Use
java.time API for parsing timestamps and dates from JSON
URL: https://github.com/apache/spark/pull/23196#discussion_r241357036
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatter.scala
##
@@ -116,29 +127,32 @@ sealed trait DateFormatter {
class Iso8601DateFormatter(
pattern: String,
-timeZone: TimeZone,
-locale: Locale) extends DateFormatter {
+locale: Locale) extends DateFormatter with FormatterUtils {
- val dateTimeFormatter = new Iso8601DateTimeFormatter(pattern, timeZone,
locale)
+ val zoneId = ZoneId.of("GMT")
Review comment:
`UTC` is a time standard, and `GMT` is `UTC+0`. So here looks like `UTC` is
more proper.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #21455: [SPARK-24093][DStream][Minor]Make some fields of KafkaStreamWriter/In…
HyukjinKwon commented on issue #21455: [SPARK-24093][DStream][Minor]Make some
fields of KafkaStreamWriter/In…
URL: https://github.com/apache/spark/pull/21455#issuecomment-446943340
BTW, fields in class constructors is not visible without `val`. For instance,
```scala
class A(fieldA: String) { }
```
generates:
```bash
$ javap -private A.class
Compiled from "A.scala"
public class A {
public A(java.lang.String);
}
```
FYI, when `val` is added:
```bash
$ javap -private A.class
Compiled from "A.scala"
public class A {
private final java.lang.String fieldA;
private java.lang.String fieldA();
public A(java.lang.String);
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23308: [SPARK-26308][SQL] Infer abstract decimal type for java/scala BigDecimal
AmplabJenkins removed a comment on issue #23308: [SPARK-26308][SQL] Infer abstract decimal type for java/scala BigDecimal URL: https://github.com/apache/spark/pull/23308#issuecomment-446942739 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100087/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23308: [SPARK-26308][SQL] Infer abstract decimal type for java/scala BigDecimal
SparkQA removed a comment on issue #23308: [SPARK-26308][SQL] Infer abstract decimal type for java/scala BigDecimal URL: https://github.com/apache/spark/pull/23308#issuecomment-446940948 **[Test build #100087 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100087/testReport)** for PR 23308 at commit [`c49329f`](https://github.com/apache/spark/commit/c49329f972841e1c400cf38d8e33065868406815). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23308: [SPARK-26308][SQL] Infer abstract decimal type for java/scala BigDecimal
AmplabJenkins removed a comment on issue #23308: [SPARK-26308][SQL] Infer abstract decimal type for java/scala BigDecimal URL: https://github.com/apache/spark/pull/23308#issuecomment-446942737 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell
HyukjinKwon commented on issue #23293: [MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell URL: https://github.com/apache/spark/pull/23293#issuecomment-446945129 Merged to master. I fixed PR description by myself. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query
AmplabJenkins removed a comment on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query URL: https://github.com/apache/spark/pull/22952#issuecomment-446948463 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6074/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query
AmplabJenkins removed a comment on issue #22952: [SPARK-20568][SS] Provide option to clean up completed files in streaming query URL: https://github.com/apache/spark/pull/22952#issuecomment-446948454 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits
AmplabJenkins removed a comment on issue #23266: [SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits URL: https://github.com/apache/spark/pull/23266#issuecomment-446908161 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100080/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
