[GitHub] [spark] MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4]
Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373960161
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr {
UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the
seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a
format that can be
- * cast to a timestamp, such as `-MM-dd` or `-MM-dd
HH:mm:ss.`
+ * cast to a timestamp, such as `-MM-dd` or `-MM-dd
HH:mm:ss`
Review comment:
Right, it works due to your changes
https://github.com/apache/spark/pull/17901 . When the `format` parameter is
omitted, `to_timestamp` uses another parsing mechanism via
`DateTimeUtils.stringToTimestamp` and the result is not truncated to seconds as
we do when the `format` is provided. Weird behavior. :-(
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #27425: [SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to StaticSQLConf
xuanyuanking commented on issue #27425: [SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to StaticSQLConf URL: https://github.com/apache/spark/pull/27425#issuecomment-581280423 Thanks for the review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
HyukjinKwon commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373957826
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr {
UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the
seconds precision.
Review comment:
Usually it's better to make the changes minimised if we have to do it to old
branches to reduce the potential conflicts when backporting. Seems this is the
case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
HyukjinKwon commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373954840
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr {
UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the
seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a
format that can be
- * cast to a timestamp, such as `-MM-dd` or `-MM-dd
HH:mm:ss.`
+ * cast to a timestamp, such as `-MM-dd` or `-MM-dd
HH:mm:ss`
Review comment:
@MaxGekk, sorry if I missed some contexts but do you mean the ones below
don't work? Seems working find in Spark 2.4.4:
```
...
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
...
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").show(false)
++
|to_timestamp('2011-01-01 12:12:12.4321')|
++
|2011-01-01 12:12:12.4321|
++
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").printSchema()
root
|-- to_timestamp('2011-01-01 12:12:12.4321'): timestamp (nullable = true)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
HyukjinKwon commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373954840
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr {
UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the
seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a
format that can be
- * cast to a timestamp, such as `-MM-dd` or `-MM-dd
HH:mm:ss.`
+ * cast to a timestamp, such as `-MM-dd` or `-MM-dd
HH:mm:ss`
Review comment:
@MaxGekk, sorry if I missed some contexts but do you mean the ones below
don't work? Seems working fine in Spark 2.4.4:
```
...
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
...
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").show(false)
++
|to_timestamp('2011-01-01 12:12:12.4321')|
++
|2011-01-01 12:12:12.4321|
++
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").printSchema()
root
|-- to_timestamp('2011-01-01 12:12:12.4321'): timestamp (nullable = true)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] peter-toth commented on issue #24553: [SPARK-27604][SQL] Enhance constant propagation
peter-toth commented on issue #24553: [SPARK-27604][SQL] Enhance constant propagation URL: https://github.com/apache/spark/pull/24553#issuecomment-581278212 @cloud-fan, @dongjoon-hyun, @maropu I improved `ConstantPropagation` a bit in this PR. Do you think you could review it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581277877 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] asfgit closed pull request #27425: [SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to StaticSQLConf
asfgit closed pull request #27425: [SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to StaticSQLConf URL: https://github.com/apache/spark/pull/27425 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581277885 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117765/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581277877 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
HyukjinKwon commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373954840
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr {
UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the
seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a
format that can be
- * cast to a timestamp, such as `-MM-dd` or `-MM-dd
HH:mm:ss.`
+ * cast to a timestamp, such as `-MM-dd` or `-MM-dd
HH:mm:ss`
Review comment:
@MaxGekk, sorry if I missed some contexts but do you mind the ones below
don't work? Seems working find in Spark 2.4.4:
```
...
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
...
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").show(false)
++
|to_timestamp('2011-01-01 12:12:12.4321')|
++
|2011-01-01 12:12:12.4321|
++
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").printSchema()
root
|-- to_timestamp('2011-01-01 12:12:12.4321'): timestamp (nullable = true)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on issue #27425: [SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to StaticSQLConf
zsxwing commented on issue #27425: [SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to StaticSQLConf URL: https://github.com/apache/spark/pull/27425#issuecomment-581276861 Thanks! Merging to master and branch-3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table
viirya commented on a change in pull request #27055: [SPARK-30394]Skip
DetermineTableStats rule when hive table can be converted to datasource table
URL: https://github.com/apache/spark/pull/27055#discussion_r373953950
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala
##
@@ -33,12 +33,10 @@ import org.apache.spark.sql.types.StructType
*
* @param sparkSession a [[SparkSession]]
* @param table the metadata of the table
- * @param sizeInBytes the table's data size in bytes
*/
class CatalogFileIndex(
sparkSession: SparkSession,
-val table: CatalogTable,
-override val sizeInBytes: Long) extends FileIndex {
Review comment:
This change as @cloud-fan said, is expensive. And it doesn't follow up the
defined behavior for partitioned data source and Hive table regrading
statistics calculation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581275959 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table
AmplabJenkins commented on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table URL: https://github.com/apache/spark/pull/27055#issuecomment-581276023 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table
AmplabJenkins removed a comment on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table URL: https://github.com/apache/spark/pull/27055#issuecomment-581276027 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22527/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581275995 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581275995 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table
AmplabJenkins commented on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table URL: https://github.com/apache/spark/pull/27055#issuecomment-581276027 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22527/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table
AmplabJenkins removed a comment on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table URL: https://github.com/apache/spark/pull/27055#issuecomment-581276023 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581275959 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581276005 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22525/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581276005 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22525/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581275972 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22526/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581275972 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22526/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581275525 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581245265 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117761/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table
SparkQA commented on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table URL: https://github.com/apache/spark/pull/27055#issuecomment-581275585 **[Test build #117764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117764/testReport)** for PR 27055 at commit [`c5297d4`](https://github.com/apache/spark/commit/c5297d48fce32b584718c04b85dd06a293bc2c2b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581275525 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
cloud-fan commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581275571 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581275530 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117763/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table
cloud-fan commented on a change in pull request #27055: [SPARK-30394]Skip
DetermineTableStats rule when hive table can be converted to datasource table
URL: https://github.com/apache/spark/pull/27055#discussion_r373952868
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala
##
@@ -61,6 +59,12 @@ class CatalogFileIndex(
override def refresh(): Unit = fileStatusCache.invalidateAll()
+ override def sizeInBytes: Long = {
+table.stats.map(_.sizeInBytes.toLong).getOrElse{
+ filterPartitions(Nil).sizeInBytes
Review comment:
This is super expensive. Are you sure we always need to do it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table
AmplabJenkins removed a comment on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table URL: https://github.com/apache/spark/pull/27055#issuecomment-569861845 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
cloud-fan commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581274793 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table
cloud-fan commented on issue #27055: [SPARK-30394]Skip DetermineTableStats rule when hive table can be converted to datasource table URL: https://github.com/apache/spark/pull/27055#issuecomment-581275046 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on issue #27185: [SPARK-30494][SQL] Avoid duplicated cached RDD when replace an existing view
cloud-fan edited a comment on issue #27185: [SPARK-30494][SQL] Avoid duplicated cached RDD when replace an existing view URL: https://github.com/apache/spark/pull/27185#issuecomment-581274289 shall we further generalize it? Currently we un-cache tables in several commands like DROP TABLE, TRUNCATE TABLE, etc. and now we find more missing places like CREATE VIEW. Instead of un-caching tables in the commands, I feel it's better to do it in low-level basic operations like `SessionCatalog.dropTable`, `createTempView`, etc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27185: [SPARK-30494][SQL] Avoid duplicated cached RDD when replace an existing view
cloud-fan commented on issue #27185: [SPARK-30494][SQL] Avoid duplicated cached RDD when replace an existing view URL: https://github.com/apache/spark/pull/27185#issuecomment-581274289 shall we further generalize it? Currently we un-cache tables in several commands like DROP TABLE, TRUNCATE TABLE, etc. and now we find more missing places like CREATE VIEW. Instead of un-caching tables in the commands, I feel it's better to do it in low-level basic operations like `SessionCatalog.dropTable`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581247012 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117762/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] fuwhu commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
fuwhu commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581273963 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp` URL: https://github.com/apache/spark/pull/27438#issuecomment-581268053 Anyway, thank you always for you active contribution. Apache Spark community really needs that.  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp` URL: https://github.com/apache/spark/pull/27438#issuecomment-581267606 @MaxGekk . Every committer's suggestion is his/her own criteria for his/her acceptance. I am also able to merge only what I can agree. And, as you know, the other committers also have different opinion and they will merge this if they agree with AS-IS status more. It always does. In addition, I don't complain about the other committer's decision when I understand it's on the edge. Since this PR is yours, it's up to you always~ @MaxGekk . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
dongjoon-hyun commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373944831
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr {
UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the
seconds precision.
Review comment:
Of course, you can do whatever you want with much more details, but I want
to minimize the change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
dongjoon-hyun commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373944890
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -2962,12 +2962,12 @@ object functions {
}
/**
- * Converts time string with the given pattern to timestamp.
+ * Converts time string with the given pattern to timestamp in the seconds
precision.
Review comment:
Same with the others.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4]
Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373942941
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -2962,12 +2962,12 @@ object functions {
}
/**
- * Converts time string with the given pattern to timestamp.
+ * Converts time string with the given pattern to timestamp in the seconds
precision.
Review comment:
Just in case, are you ok with this change?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4]
Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373942625
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr {
UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the
seconds precision.
Review comment:
May I ask you to explain why we shouldn't clarify function's behavior here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp` URL: https://github.com/apache/spark/pull/27438#issuecomment-581262355 In general, this seems to be a follow-up of SPARK-23792 (at 2.4.0). Since it's too old JIRA, we had better use another JIRA or use `[MINOR]` like you. I support your choice. I believe that one thing we need is that `-MM-dd HH:mm:ss.` -> `-MM-dd HH:mm:ss` change. I recommend to focus on the above change only. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #21888: [SPARK-24253][SQL][WIP] Implement DeleteFrom for v2 tables
cloud-fan commented on issue #21888: [SPARK-24253][SQL][WIP] Implement DeleteFrom for v2 tables URL: https://github.com/apache/spark/pull/21888#issuecomment-581261506 This is done by https://github.com/apache/spark/pull/25115, closing this PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #21888: [SPARK-24253][SQL][WIP] Implement DeleteFrom for v2 tables
cloud-fan closed pull request #21888: [SPARK-24253][SQL][WIP] Implement DeleteFrom for v2 tables URL: https://github.com/apache/spark/pull/21888 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
dongjoon-hyun commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp` URL: https://github.com/apache/spark/pull/27438#discussion_r373940008 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -1370,7 +1370,7 @@ case class ParseToDate(left: Expression, format: Option[Expression], child: Expr } /** - * Parses a column to a timestamp based on the supplied format. + * Parses a column to a timestamp in the seconds precision based on the supplied format. Review comment: Let's not touch this file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
dongjoon-hyun commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373940062
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr {
UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the
seconds precision.
Review comment:
Let's not touch this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2)
AmplabJenkins removed a comment on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2) URL: https://github.com/apache/spark/pull/27439#issuecomment-581254453 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117756/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2)
AmplabJenkins removed a comment on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2) URL: https://github.com/apache/spark/pull/27439#issuecomment-58125 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2)
SparkQA removed a comment on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2) URL: https://github.com/apache/spark/pull/27439#issuecomment-581210072 **[Test build #117756 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117756/testReport)** for PR 27439 at commit [`b792307`](https://github.com/apache/spark/commit/b792307cf3d8bd09779922bd390367b5abfd83ee). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2)
AmplabJenkins commented on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2) URL: https://github.com/apache/spark/pull/27439#issuecomment-58125 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27423: [SPARK-30697][SQL] Handle database and namespace exceptions in catalog.isView
cloud-fan closed pull request #27423: [SPARK-30697][SQL] Handle database and namespace exceptions in catalog.isView URL: https://github.com/apache/spark/pull/27423 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2)
AmplabJenkins commented on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2) URL: https://github.com/apache/spark/pull/27439#issuecomment-581254453 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117756/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27423: [SPARK-30697][SQL] Handle database and namespace exceptions in catalog.isView
cloud-fan commented on issue #27423: [SPARK-30697][SQL] Handle database and namespace exceptions in catalog.isView URL: https://github.com/apache/spark/pull/27423#issuecomment-581254173 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2)
SparkQA commented on issue #27439: [SPARK-29108][SQL][TESTS][FOLLOWUP] Add insert sql to window.sql (Part 2) URL: https://github.com/apache/spark/pull/27439#issuecomment-581253978 **[Test build #117756 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117756/testReport)** for PR 27439 at commit [`b792307`](https://github.com/apache/spark/commit/b792307cf3d8bd09779922bd390367b5abfd83ee). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
Ngone51 commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581249081 @dongjoon-hyun Would you please help check Jenkins failure? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581247278 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22524/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581247271 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581247271 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581247278 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22524/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins removed a comment on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581247010 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581247012 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117762/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning
AmplabJenkins commented on issue #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning URL: https://github.com/apache/spark/pull/27232#issuecomment-581247010 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27223: [SPARK-30511][SPARK-28403][CORE] Don't treat failed/killed speculative tasks as pending in ExecutorAllocationManager
Ngone51 commented on a change in pull request #27223:
[SPARK-30511][SPARK-28403][CORE] Don't treat failed/killed speculative tasks as
pending in ExecutorAllocationManager
URL: https://github.com/apache/spark/pull/27223#discussion_r373928758
##
File path: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
##
@@ -614,18 +615,30 @@ private[spark] class ExecutorAllocationManager(
stageAttemptToNumRunningTask -= stageAttempt
}
}
-// If the task failed, we expect it to be resubmitted later. To ensure
we have
-// enough resources to run the resubmitted task, we need to mark the
scheduler
-// as backlogged again if it's not already marked as such (SPARK-8366)
-if (taskEnd.reason != Success) {
- if (totalPendingTasks() == 0) {
-allocationManager.onSchedulerBacklogged()
- }
- if (taskEnd.taskInfo.speculative) {
-stageAttemptToSpeculativeTaskIndices.get(stageAttempt).foreach
{_.remove(taskIndex)}
- } else {
-stageAttemptToTaskIndices.get(stageAttempt).foreach
{_.remove(taskIndex)}
- }
+
+if (taskEnd.taskInfo.speculative) {
+ stageAttemptToSpeculativeTaskIndices.get(stageAttempt).foreach
{_.remove{taskIndex}}
+ stageAttemptToNumSpeculativeTasks(stageAttempt) -= 1
+}
+
+taskEnd.reason match {
+ case Success | _: TaskKilled =>
+ case _ =>
+if (totalPendingTasks() == 0) {
+ // If the task failed (not intentionally killed), we expect it
to be resubmitted
+ // later. To ensure we have enough resources to run the
resubmitted task, we need to
+ // mark the scheduler as backlogged again if it's not already
marked as such
+ // (SPARK-8366)
+ allocationManager.onSchedulerBacklogged()
Review comment:
make sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] fuwhu commented on a change in pull request #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition prunin
fuwhu commented on a change in pull request #27232:
[SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after
pushing down to SessionCatalog for partition pruning
URL: https://github.com/apache/spark/pull/27232#discussion_r373928650
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScanExec.scala
##
@@ -162,11 +162,29 @@ case class HiveTableScanExec(
}
}
+ @transient lazy val prunedPartitions: Seq[HivePartition] = {
+if (relation.prunedPartitions.nonEmpty) {
+ val hivePartitions =
+relation.prunedPartitions.get.map(HiveClientImpl.toHivePartition(_,
hiveQlTable))
+ if (partitionPruningPred.forall(!ExecSubqueryExpression.hasSubquery(_)))
{
+hivePartitions
+ } else {
+prunePartitions(hivePartitions)
+ }
+} else {
+ if (sparkSession.sessionState.conf.metastorePartitionPruning) {
+rawPartitions
+ } else {
+prunePartitions(rawPartitions)
+ }
+}
+ }
+
// exposed for tests
- @transient lazy val rawPartitions = {
+ @transient lazy val rawPartitions: Seq[HivePartition] = {
Review comment:
sure, removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] fuwhu commented on a change in pull request #27232: [SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition prunin
fuwhu commented on a change in pull request #27232:
[SPARK-30525][SQL]HiveTableScanExec do not need to prune partitions again after
pushing down to SessionCatalog for partition pruning
URL: https://github.com/apache/spark/pull/27232#discussion_r373928619
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScanExec.scala
##
@@ -162,11 +162,29 @@ case class HiveTableScanExec(
}
}
+ @transient lazy val prunedPartitions: Seq[HivePartition] = {
+if (relation.prunedPartitions.nonEmpty) {
+ val hivePartitions =
+relation.prunedPartitions.get.map(HiveClientImpl.toHivePartition(_,
hiveQlTable))
+ if (partitionPruningPred.forall(!ExecSubqueryExpression.hasSubquery(_)))
{
+hivePartitions
+ } else {
+prunePartitions(hivePartitions)
+ }
+} else {
+ if (sparkSession.sessionState.conf.metastorePartitionPruning) {
Review comment:
sure, added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581245656 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27423: [SPARK-30697][SQL] Handle database and namespace exceptions in catalog.isView
cloud-fan commented on a change in pull request #27423: [SPARK-30697][SQL]
Handle database and namespace exceptions in catalog.isView
URL: https://github.com/apache/spark/pull/27423#discussion_r373927979
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##
@@ -826,6 +826,8 @@ class SessionCatalog(
getTempViewOrPermanentTableMetadata(ident).tableType ==
CatalogTableType.VIEW
} catch {
case _: NoSuchTableException => false
+case _: NoSuchDatabaseException => false
+case _: NoSuchNamespaceException => false
Review comment:
good catch!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581245658 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22523/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581245656 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581245658 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22523/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581245256 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581245265 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117761/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581245256 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] linzebing commented on a change in pull request #27223: [SPARK-30511][SPARK-28403][CORE] Don't treat failed/killed speculative tasks as pending in ExecutorAllocationManager
linzebing commented on a change in pull request #27223:
[SPARK-30511][SPARK-28403][CORE] Don't treat failed/killed speculative tasks as
pending in ExecutorAllocationManager
URL: https://github.com/apache/spark/pull/27223#discussion_r373927513
##
File path: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
##
@@ -614,18 +615,30 @@ private[spark] class ExecutorAllocationManager(
stageAttemptToNumRunningTask -= stageAttempt
}
}
-// If the task failed, we expect it to be resubmitted later. To ensure
we have
-// enough resources to run the resubmitted task, we need to mark the
scheduler
-// as backlogged again if it's not already marked as such (SPARK-8366)
-if (taskEnd.reason != Success) {
- if (totalPendingTasks() == 0) {
-allocationManager.onSchedulerBacklogged()
- }
- if (taskEnd.taskInfo.speculative) {
-stageAttemptToSpeculativeTaskIndices.get(stageAttempt).foreach
{_.remove(taskIndex)}
- } else {
-stageAttemptToTaskIndices.get(stageAttempt).foreach
{_.remove(taskIndex)}
- }
+
+if (taskEnd.taskInfo.speculative) {
+ stageAttemptToSpeculativeTaskIndices.get(stageAttempt).foreach
{_.remove{taskIndex}}
+ stageAttemptToNumSpeculativeTasks(stageAttempt) -= 1
+}
+
+taskEnd.reason match {
+ case Success | _: TaskKilled =>
+ case _ =>
+if (totalPendingTasks() == 0) {
+ // If the task failed (not intentionally killed), we expect it
to be resubmitted
+ // later. To ensure we have enough resources to run the
resubmitted task, we need to
+ // mark the scheduler as backlogged again if it's not already
marked as such
+ // (SPARK-8366)
+ allocationManager.onSchedulerBacklogged()
Review comment:
The speculation interval is default to 100ms, so a new speculative task will
launched almost instantly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581241640 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117760/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
Ngone51 commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581244605 Jenkins, retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27223: [SPARK-30511][SPARK-28403][CORE] Don't treat failed/killed speculative tasks as pending in ExecutorAllocationManager
Ngone51 commented on a change in pull request #27223:
[SPARK-30511][SPARK-28403][CORE] Don't treat failed/killed speculative tasks as
pending in ExecutorAllocationManager
URL: https://github.com/apache/spark/pull/27223#discussion_r373926875
##
File path: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
##
@@ -614,18 +615,30 @@ private[spark] class ExecutorAllocationManager(
stageAttemptToNumRunningTask -= stageAttempt
}
}
-// If the task failed, we expect it to be resubmitted later. To ensure
we have
-// enough resources to run the resubmitted task, we need to mark the
scheduler
-// as backlogged again if it's not already marked as such (SPARK-8366)
-if (taskEnd.reason != Success) {
- if (totalPendingTasks() == 0) {
-allocationManager.onSchedulerBacklogged()
- }
- if (taskEnd.taskInfo.speculative) {
-stageAttemptToSpeculativeTaskIndices.get(stageAttempt).foreach
{_.remove(taskIndex)}
- } else {
-stageAttemptToTaskIndices.get(stageAttempt).foreach
{_.remove(taskIndex)}
- }
+
+if (taskEnd.taskInfo.speculative) {
+ stageAttemptToSpeculativeTaskIndices.get(stageAttempt).foreach
{_.remove{taskIndex}}
+ stageAttemptToNumSpeculativeTasks(stageAttempt) -= 1
+}
+
+taskEnd.reason match {
+ case Success | _: TaskKilled =>
+ case _ =>
+if (totalPendingTasks() == 0) {
+ // If the task failed (not intentionally killed), we expect it
to be resubmitted
+ // later. To ensure we have enough resources to run the
resubmitted task, we need to
+ // mark the scheduler as backlogged again if it's not already
marked as such
+ // (SPARK-8366)
+ allocationManager.onSchedulerBacklogged()
Review comment:
> If a speculative task fails, while it will not be directly resubmitted, a
new speculative task will be launched in next speculation cycle. So it's OK for
us to mark the scheduler as backlogged in this case.
Hmm...but the new speculative task may not be launched if the normal task
finish. And even if it launched, `ExecutorAllocationManager` could still handle
it by receiving `SparkListenerSpeculativeTaskSubmitted`. Though, calling
`onSchedulerBacklogged` may could reserve executor resource more early to
reduce the delay. Fine!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581242112 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581242119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22522/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581242119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22522/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581242112 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581241638 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581238204 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117759/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581241640 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117760/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581241638 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] linzebing commented on a change in pull request #27223: [SPARK-30511][SPARK-28403][CORE] Don't treat failed/killed speculative tasks as pending in ExecutorAllocationManager
linzebing commented on a change in pull request #27223:
[SPARK-30511][SPARK-28403][CORE] Don't treat failed/killed speculative tasks as
pending in ExecutorAllocationManager
URL: https://github.com/apache/spark/pull/27223#discussion_r373924312
##
File path: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
##
@@ -614,18 +615,30 @@ private[spark] class ExecutorAllocationManager(
stageAttemptToNumRunningTask -= stageAttempt
}
}
-// If the task failed, we expect it to be resubmitted later. To ensure
we have
-// enough resources to run the resubmitted task, we need to mark the
scheduler
-// as backlogged again if it's not already marked as such (SPARK-8366)
-if (taskEnd.reason != Success) {
- if (totalPendingTasks() == 0) {
-allocationManager.onSchedulerBacklogged()
- }
- if (taskEnd.taskInfo.speculative) {
-stageAttemptToSpeculativeTaskIndices.get(stageAttempt).foreach
{_.remove(taskIndex)}
- } else {
-stageAttemptToTaskIndices.get(stageAttempt).foreach
{_.remove(taskIndex)}
- }
+
+if (taskEnd.taskInfo.speculative) {
+ stageAttemptToSpeculativeTaskIndices.get(stageAttempt).foreach
{_.remove{taskIndex}}
+ stageAttemptToNumSpeculativeTasks(stageAttempt) -= 1
+}
+
+taskEnd.reason match {
+ case Success | _: TaskKilled =>
+ case _ =>
+if (totalPendingTasks() == 0) {
+ // If the task failed (not intentionally killed), we expect it
to be resubmitted
+ // later. To ensure we have enough resources to run the
resubmitted task, we need to
+ // mark the scheduler as backlogged again if it's not already
marked as such
+ // (SPARK-8366)
+ allocationManager.onSchedulerBacklogged()
Review comment:
As explained in the comment, if a task is intentionally killed, we should
not mark the scheduler as backlogged.
If a speculative task fails, while it will not be directly resubmitted, a
new speculative task will be launched in next speculation cycle. So it's OK for
us to mark the scheduler as backlogged in this case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581238529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22521/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581238529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22521/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581238525 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581238525 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins removed a comment on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581238198 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581238204 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117759/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false
AmplabJenkins commented on issue #27306: [SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false URL: https://github.com/apache/spark/pull/27306#issuecomment-581238198 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz edited a comment on issue #27391: [SPARK-30612][SQL] Resolve qualified column name with v2 tables
brkyvz edited a comment on issue #27391: [SPARK-30612][SQL] Resolve qualified column name with v2 tables URL: https://github.com/apache/spark/pull/27391#issuecomment-581237957 Agree with @cloud-fan. Please do not remove the old code in resolution for the following too reasons: 1. It's very risky to do so right now 2. '# of qualifiers < 3' will still be the most common case. We should still have the proper resolution there. I think your matcher should come in if the first two qualifier matching code doesn't work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz edited a comment on issue #27391: [SPARK-30612][SQL] Resolve qualified column name with v2 tables
brkyvz edited a comment on issue #27391: [SPARK-30612][SQL] Resolve qualified column name with v2 tables URL: https://github.com/apache/spark/pull/27391#issuecomment-581237957 Agree with @cloud-fan. Please do not remove the old code in resolution for the following too reasons: 1. It's very risky to do so right now 2. '# of qualifiers < 3' will still be the most common case. We should still have the proper resolution there. I think your matcher should come in if the first two qualifier matching code doesn't work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on issue #27391: [SPARK-30612][SQL] Resolve qualified column name with v2 tables
brkyvz commented on issue #27391: [SPARK-30612][SQL] Resolve qualified column name with v2 tables URL: https://github.com/apache/spark/pull/27391#issuecomment-581237957 Agree with @cloud-fan. Please do not remove the old code in resolution for the following too reasons: 1. It's very risky to do so right now 2. # of qualifiers < 3 will still be the most common case. We should still have the proper resolution there. I think your matcher should come in if the first two qualifier matching code doesn't work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
