[GitHub] [spark] cloud-fan commented on a diff in pull request #37831: [SPARK-40354][SQL] Support eliminate dynamic partition for datasource v1 writes

cloud-fan commented on code in PR #37831: URL: https://github.com/apache/spark/pull/37831#discussion_r966713024 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala: ## @@ -477,7 +480,7 @@ case class DataSource( // will be adjusted within

[GitHub] [spark] SparksFyz commented on pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`

SparksFyz commented on PR #37825: URL: https://github.com/apache/spark/pull/37825#issuecomment-1241662233 https://github.com/apache/spark/pull/34953 take a look this pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] steveloughran commented on pull request #37468: [SPARK-40034][SQL] PathOutputCommitters to support dynamic partitions

steveloughran commented on PR #37468: URL: https://github.com/apache/spark/pull/37468#issuecomment-1241751151 @dongjoon-hyun I'm off on vacation next week; we will fork off the branch the week after. things i'd like in *if anyone has the time* 1. upgraded shaded parquet 2. the

[GitHub] [spark] peter-toth commented on a diff in pull request #36027: [SPARK-38717][SQL] Handle Hive's bucket spec case preserving behaviour

peter-toth commented on code in PR #36027: URL: https://github.com/apache/spark/pull/36027#discussion_r966850639 ## sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala: ## @@ -1095,7 +1095,11 @@ private[hive] object HiveClientImpl extends Logging {

[GitHub] [spark] peter-toth commented on a diff in pull request #36027: [SPARK-38717][SQL] Handle Hive's bucket spec case preserving behaviour

peter-toth commented on code in PR #36027: URL: https://github.com/apache/spark/pull/36027#discussion_r966850639 ## sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala: ## @@ -1095,7 +1095,11 @@ private[hive] object HiveClientImpl extends Logging {

[GitHub] [spark] cloud-fan commented on a diff in pull request #37841: [SPARK-40324][SQL] Provide query context in AnalysisException

cloud-fan commented on code in PR #37841: URL: https://github.com/apache/spark/pull/37841#discussion_r966744144 ## sql/catalyst/src/main/scala/org/apache/spark/sql/AnalysisException.scala: ## @@ -124,12 +126,16 @@ class AnalysisException protected[sql] ( plan:

[GitHub] [spark] ulysses-you commented on a diff in pull request #37831: [SPARK-40354][SQL] Support eliminate dynamic partition for datasource v1 writes

ulysses-you commented on code in PR #37831: URL: https://github.com/apache/spark/pull/37831#discussion_r966786913 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala: ## @@ -477,7 +480,7 @@ case class DataSource( // will be adjusted

[GitHub] [spark] cloud-fan commented on a diff in pull request #37831: [SPARK-40354][SQL] Support eliminate dynamic partition for datasource v1 writes

cloud-fan commented on code in PR #37831: URL: https://github.com/apache/spark/pull/37831#discussion_r966804738 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala: ## @@ -477,7 +480,7 @@ case class DataSource( // will be adjusted within

[GitHub] [spark] cloud-fan commented on a diff in pull request #36850: [SPARK-39069][SQL] Enhance ConstantPropagation to replace constants in inequality predicates

cloud-fan commented on code in PR #36850: URL: https://github.com/apache/spark/pull/36850#discussion_r966808655 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala: ## @@ -205,10 +205,14 @@ object ConstantPropagation extends

[GitHub] [spark] martin-g commented on a diff in pull request #37844: [DON'T MERGE][BUILD] Upgrade slf4j to 2.0.0

martin-g commented on code in PR #37844: URL: https://github.com/apache/spark/pull/37844#discussion_r966812989 ## pom.xml: ## @@ -112,7 +112,7 @@ 3.8.6 1.6.0 spark -1.7.36 +2.0.0 2.18.0 Review Comment: Log4j 2.18.0 does not support SLF4J 2.x. See

[GitHub] [spark] cloud-fan commented on pull request #37824: [SPARK-40362][SQL] Bug in Canonicalization of expressions like Add & Multiply i.e Commutative Operators

cloud-fan commented on PR #37824: URL: https://github.com/apache/spark/pull/37824#issuecomment-1241713531 Agree with @peter-toth that we should do all ordering in the 2nd pass, in a bottom-up way. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] cloud-fan commented on a diff in pull request #37830: [SPARK-40387][SQL] Improve the implementation of Spark Decimal

cloud-fan commented on code in PR #37830: URL: https://github.com/apache/spark/pull/37830#discussion_r966829687 ## sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala: ## @@ -504,7 +505,7 @@ final class Decimal extends Ordered[Decimal] with Serializable {

[GitHub] [spark] cloud-fan commented on a diff in pull request #37841: [SPARK-40324][SQL] Provide query context in AnalysisException

cloud-fan commented on code in PR #37841: URL: https://github.com/apache/spark/pull/37841#discussion_r966738071 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ExpressionTypeCheckingSuite.scala: ## @@ -77,12 +77,6 @@ class ExpressionTypeCheckingSuite

[GitHub] [spark] cloud-fan commented on a diff in pull request #37830: [SPARK-40387][SQL] Improve the implementation of Spark Decimal

cloud-fan commented on code in PR #37830: URL: https://github.com/apache/spark/pull/37830#discussion_r966747029 ## sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala: ## @@ -184,68 +184,56 @@ final class Decimal extends Ordered[Decimal] with Serializable {

[GitHub] [spark] AmplabJenkins commented on pull request #37822: [SPARK-40381][DEPLOY] Support standalone worker recommission

AmplabJenkins commented on PR #37822: URL: https://github.com/apache/spark/pull/37822#issuecomment-1241676197 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins commented on pull request #37824: [SPARK-40362][SQL] Bug in Canonicalization of expressions like Add & Multiply i.e Commutative Operators

AmplabJenkins commented on PR #37824: URL: https://github.com/apache/spark/pull/37824#issuecomment-1241676122 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] zhengruifeng opened a new pull request, #37845: [SPARK-40399][PS] DataFrame.corr `Pearson` support missing values and `min_periods `

zhengruifeng opened a new pull request, #37845: URL: https://github.com/apache/spark/pull/37845 ### What changes were proposed in this pull request? refactor `pearson` correlation in `DataFrame.corr` to: 1, support missing values; 2, add parameter `min_periods`; 3, enable arrow

[GitHub] [spark] cloud-fan commented on a diff in pull request #36027: [SPARK-38717][SQL] Handle Hive's bucket spec case preserving behaviour

cloud-fan commented on code in PR #36027: URL: https://github.com/apache/spark/pull/36027#discussion_r966801663 ## sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala: ## @@ -1095,7 +1095,11 @@ private[hive] object HiveClientImpl extends Logging {

[GitHub] [spark] peter-toth commented on a diff in pull request #36027: [SPARK-38717][SQL] Handle Hive's bucket spec case preserving behaviour

peter-toth commented on code in PR #36027: URL: https://github.com/apache/spark/pull/36027#discussion_r966809657 ## sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala: ## @@ -1095,7 +1095,11 @@ private[hive] object HiveClientImpl extends Logging {

[GitHub] [spark] peter-toth commented on a diff in pull request #36027: [SPARK-38717][SQL] Handle Hive's bucket spec case preserving behaviour

peter-toth commented on code in PR #36027: URL: https://github.com/apache/spark/pull/36027#discussion_r966809657 ## sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala: ## @@ -1095,7 +1095,11 @@ private[hive] object HiveClientImpl extends Logging {

[GitHub] [spark] LuciferYang commented on a diff in pull request #37844: [DON'T MERGE][BUILD] Upgrade slf4j to 2.0.0

LuciferYang commented on code in PR #37844: URL: https://github.com/apache/spark/pull/37844#discussion_r966814386 ## pom.xml: ## @@ -112,7 +112,7 @@ 3.8.6 1.6.0 spark -1.7.36 +2.0.0 2.18.0 Review Comment: Thanks @martin-g ~ -- This is an

[GitHub] [spark] LuciferYang commented on pull request #37844: [DON'T MERGE][BUILD] Upgrade slf4j to 2.0.0

LuciferYang commented on PR #37844: URL: https://github.com/apache/spark/pull/37844#issuecomment-1241715672 close and wait log4j 2.19.x -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] LuciferYang closed pull request #37844: [DON'T MERGE][BUILD] Upgrade slf4j to 2.0.0

LuciferYang closed pull request #37844: [DON'T MERGE][BUILD] Upgrade slf4j to 2.0.0 URL: https://github.com/apache/spark/pull/37844 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] cloud-fan commented on a diff in pull request #37830: [SPARK-40387][SQL] Improve the implementation of Spark Decimal

cloud-fan commented on code in PR #37830: URL: https://github.com/apache/spark/pull/37830#discussion_r966827047 ## sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala: ## @@ -240,9 +240,11 @@ final class Decimal extends Ordered[Decimal] with Serializable {

[GitHub] [spark] zzcclp opened a new pull request, #37846: [SPARK-40280][SQL][FOLLOWUP][3.2] Fix 'ParquetFilterSuite' issue

zzcclp opened a new pull request, #37846: URL: https://github.com/apache/spark/pull/37846 ### What changes were proposed in this pull request? Fix 'ParquetFilterSuite' issue after merging #37747 : The `org.apache.parquet.filter2.predicate.Operators.In` was added in the

[GitHub] [spark] zzcclp opened a new pull request, #37847: [SPARK-40280][SQL][FOLLOWUP][3.3] Fix 'ParquetFilterSuite' issue

zzcclp opened a new pull request, #37847: URL: https://github.com/apache/spark/pull/37847 ### What changes were proposed in this pull request? ### Why are the changes needed? Fix 'ParquetFilterSuite' issue after merging #37747 : The

[GitHub] [spark] zzcclp commented on pull request #37847: [SPARK-40280][SQL][FOLLOWUP][3.3] Fix 'ParquetFilterSuite' issue

zzcclp commented on PR #37847: URL: https://github.com/apache/spark/pull/37847#issuecomment-1241948916 @revans2 @cloud-fan @wangyum please take a review, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] zzcclp commented on pull request #37846: [SPARK-40280][SQL][FOLLOWUP][3.2] Fix 'ParquetFilterSuite' issue

zzcclp commented on PR #37846: URL: https://github.com/apache/spark/pull/37846#issuecomment-1241948809 @revans2 @cloud-fan @wangyum please take a review, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] tgravescs commented on a diff in pull request #37826: [SPARK-40364][CORE] Use the unified `DBProvider#initDB ` method

tgravescs commented on code in PR #37826: URL: https://github.com/apache/spark/pull/37826#discussion_r967137328 ## common/network-common/src/main/java/org/apache/spark/network/util/LevelDBProvider.java: ## @@ -85,14 +84,6 @@ public static DB initLevelDB(File dbFile,

[GitHub] [spark] allisonwang-db commented on a diff in pull request #37840: [SPARK-40394][SQL] Move subquery expression CheckAnalysis error messages to use the new error framework

allisonwang-db commented on code in PR #37840: URL: https://github.com/apache/spark/pull/37840#discussion_r967150145 ## core/src/main/resources/error/error-classes.json: ## @@ -327,6 +327,83 @@ ], "sqlState" : "42000" }, + "INVALID_SUBQUERY_EXPRESSION" : { +

<    1   2