[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960245752 ## project/MimaExcludes.scala: ## @@ -118,79 +96,23 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] cloud-fan commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules

cloud-fan commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960244646 ## project/MimaExcludes.scala: ## @@ -118,79 +96,23 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] linhongliu-db commented on pull request #37742: [SPARK-40291][SQL] Improve the message for column not in group by clause error

linhongliu-db commented on PR #37742: URL: https://github.com/apache/spark/pull/37742#issuecomment-1233770066 cc @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] linhongliu-db commented on pull request #37742: [SPARK-40291][SQL] Improve the message for column not in group by clause error

linhongliu-db commented on PR #37742: URL: https://github.com/apache/spark/pull/37742#issuecomment-1233769917 cc @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] dongjoon-hyun closed pull request #37622: [SPARK-40187][DOCS] Add `Apache YuniKorn` scheduler docs

dongjoon-hyun closed pull request #37622: [SPARK-40187][DOCS] Add `Apache YuniKorn` scheduler docs URL: https://github.com/apache/spark/pull/37622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] dongjoon-hyun commented on pull request #37622: [SPARK-40187][DOCS] Add `Apache YuniKorn` scheduler docs

dongjoon-hyun commented on PR #37622: URL: https://github.com/apache/spark/pull/37622#issuecomment-1233757469 Sorry for being late, @yangwwei . We can move forward more based on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960229821 ## project/MimaExcludes.scala: ## @@ -118,79 +96,23 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960227179 ## project/MimaExcludes.scala: ## @@ -118,79 +96,23 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] beliefer commented on pull request #37736: [SPARK-40285][SQL] Simplify the `roundTo[Numeric]` for Spark `Decimal`

beliefer commented on PR #37736: URL: https://github.com/apache/spark/pull/37736#issuecomment-1233738789 @cloud-fan Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] aokolnychyi commented on a diff in pull request #37749: [SPARK-40295][SQL] Allow v2 functions with literal args in write distribution/ordering

aokolnychyi commented on code in PR #37749: URL: https://github.com/apache/spark/pull/37749#discussion_r960216556 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala: ## @@ -105,18 +105,27 @@ object V2ExpressionUtils extends

[GitHub] [spark] cloud-fan commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules

cloud-fan commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960214675 ## project/MimaExcludes.scala: ## @@ -118,79 +96,23 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] cloud-fan commented on a diff in pull request #37746: [SPARK-40293][SQL] Make the V2 table error message more meaningful

cloud-fan commented on code in PR #37746: URL: https://github.com/apache/spark/pull/37746#discussion_r960213508 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -542,7 +542,9 @@ private[sql] object QueryCompilationErrors extends

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960201922 ## project/MimaExcludes.scala: ## @@ -118,79 +133,24 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960213062 ## project/MimaExcludes.scala: ## @@ -62,11 +62,26 @@ object MimaExcludes {

[GitHub] [spark] cloud-fan closed pull request #37736: [SPARK-40285][SQL] Simplify the `roundTo[Numeric]` for Spark `Decimal`

cloud-fan closed pull request #37736: [SPARK-40285][SQL] Simplify the `roundTo[Numeric]` for Spark `Decimal` URL: https://github.com/apache/spark/pull/37736 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] cloud-fan commented on pull request #37736: [SPARK-40285][SQL] Simplify the `roundTo[Numeric]` for Spark `Decimal`

cloud-fan commented on PR #37736: URL: https://github.com/apache/spark/pull/37736#issuecomment-1233732666 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960210967 ## project/MimaExcludes.scala: ## @@ -118,79 +96,23 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] cloud-fan commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

cloud-fan commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960209742 ## project/MimaExcludes.scala: ## @@ -118,79 +96,23 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] aokolnychyi commented on a diff in pull request #37749: [SPARK-40295][SQL] Allow v2 functions with literal args in write distribution/ordering

aokolnychyi commented on code in PR #37749: URL: https://github.com/apache/spark/pull/37749#discussion_r960207282 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala: ## @@ -105,18 +105,27 @@ object V2ExpressionUtils extends

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960206275 ## project/MimaExcludes.scala: ## @@ -118,79 +133,24 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960205832 ## project/MimaExcludes.scala: ## @@ -118,79 +133,24 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960202573 ## project/MimaExcludes.scala: ## @@ -62,11 +62,26 @@ object MimaExcludes {

[GitHub] [spark] 1zg12 commented on pull request #37738: add Support Java Class with circular references

1zg12 commented on PR #37738: URL: https://github.com/apache/spark/pull/37738#issuecomment-1233718230 > Can you describe a valid use case? I can't think of one. Encoders are used with data classes, bean-like classes Google Protobuf is an example, it's widely used as a data class. In

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960201922 ## project/MimaExcludes.scala: ## @@ -118,79 +133,24 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

dongjoon-hyun commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960199402 ## project/MimaExcludes.scala: ## @@ -118,79 +133,24 @@ object MimaExcludes { ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

dongjoon-hyun commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960198725 ## project/MimaExcludes.scala: ## @@ -62,11 +62,26 @@ object MimaExcludes {

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

dongjoon-hyun commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960197251 ## project/MimaExcludes.scala: ## @@ -102,8 +157,8 @@ object MimaExcludes {

[GitHub] [spark] srowen commented on pull request #37738: add Support Java Class with circular references

srowen commented on PR #37738: URL: https://github.com/apache/spark/pull/37738#issuecomment-1233698787 Can you describe a valid use case? I can't think of one. Encoders are used with data classes, bean-like classes -- This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960190241 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] 1zg12 commented on pull request #37738: add Support Java Class with circular references

1zg12 commented on PR #37738: URL: https://github.com/apache/spark/pull/37738#issuecomment-1233696662 > Hm, skipping them doesn't seem right either. Not sure if this should be an option; it is just something that doesn't make sense to encode If it's a field the developer/application

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960187731 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960187254 ## project/MimaExcludes.scala: ## @@ -102,8 +157,8 @@ object MimaExcludes {

[GitHub] [spark] AmplabJenkins commented on pull request #37723: Queries can see group by result

AmplabJenkins commented on PR #37723: URL: https://github.com/apache/spark/pull/37723#issuecomment-1233693460 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960186703 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] cloud-fan commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

cloud-fan commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960186203 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960186185 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960185778 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960182874 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960184187 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960185199 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] cloud-fan commented on a diff in pull request #37751: [SPARK-40297][SQL] CTE outer reference nested in CTE main body cannot be resolved

cloud-fan commented on code in PR #37751: URL: https://github.com/apache/spark/pull/37751#discussion_r960184855 ## sql/core/src/test/resources/sql-tests/inputs/cte-nested.sql: ## @@ -146,4 +146,61 @@ WITH ) SELECT * FROM t3 ) -SELECT * FROM t2; \ No newline at end

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960184187 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960184187 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960184187 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] zhengruifeng commented on pull request #37748: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call

zhengruifeng commented on PR #37748: URL: https://github.com/apache/spark/pull/37748#issuecomment-1233689711 `pmod` was newly added in https://github.com/apache/spark/commit/f8b15395cf347b6c6c6a4a20077fdeb31bfabb24 do we need to make it support the legacy? -- This is an automated

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960184187 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] beliefer commented on pull request #37736: [SPARK-40285][SQL] Simplify the `roundTo[Numeric]` for Spark `Decimal`

beliefer commented on PR #37736: URL: https://github.com/apache/spark/pull/37736#issuecomment-1233689147 ping @MaxGekk @dongjoon-hyun @gengliangwang @viirya @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960182874 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960182874 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960182874 ## project/MimaExcludes.scala: ## @@ -62,11 +62,66 @@ object MimaExcludes {

[GitHub] [spark] huaxingao commented on pull request #37746: [SPARK-40293][SQL] Make the V2 table error message more meaningful

huaxingao commented on PR #37746: URL: https://github.com/apache/spark/pull/37746#issuecomment-1233679830 cc @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] cloud-fan closed pull request #37737: [SPARK-40055][SQL][FOLLOWUP] CatalogManager.listCatalogs should include spark_catalog

cloud-fan closed pull request #37737: [SPARK-40055][SQL][FOLLOWUP] CatalogManager.listCatalogs should include spark_catalog URL: https://github.com/apache/spark/pull/37737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] cloud-fan commented on pull request #37737: [SPARK-40055][SQL][FOLLOWUP] CatalogManager.listCatalogs should include spark_catalog

cloud-fan commented on PR #37737: URL: https://github.com/apache/spark/pull/37737#issuecomment-1233675420 thanks for review, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] maryannxue commented on pull request #37751: [SPARK-40297][SQL] CTE outer reference nested in CTE main body cannot be resolved

maryannxue commented on PR #37751: URL: https://github.com/apache/spark/pull/37751#issuecomment-1233675472 cc @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] maryannxue opened a new pull request, #37751: [SPARK-40297][SQL] CTE outer reference nested in CTE main body cannot be resolved

maryannxue opened a new pull request, #37751: URL: https://github.com/apache/spark/pull/37751 ### What changes were proposed in this pull request? This PR fixes a bug where a CTE reference cannot be resolved if this reference occurs in an inner CTE definition nested in the

[GitHub] [spark] cloud-fan commented on a diff in pull request #37520: [SPARK-40098][SQL] Format error messages in the Thrift Server

cloud-fan commented on code in PR #37520: URL: https://github.com/apache/spark/pull/37520#discussion_r960169126 ## sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServerErrors.scala: ## @@ -36,11 +36,10 @@ object HiveThriftServerErrors {

[GitHub] [spark] cloud-fan commented on a diff in pull request #37407: [SPARK-39876][SQL] Add UNPIVOT to SQL syntax

cloud-fan commented on code in PR #37407: URL: https://github.com/apache/spark/pull/37407#discussion_r960164704 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala: ## @@ -1375,28 +1375,84 @@ case class Pivot( * A constructor

[GitHub] [spark] cloud-fan commented on a diff in pull request #37407: [SPARK-39876][SQL] Add UNPIVOT to SQL syntax

cloud-fan commented on code in PR #37407: URL: https://github.com/apache/spark/pull/37407#discussion_r960164566 ## sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4: ## @@ -618,6 +618,46 @@ pivotValue : expression (AS? identifier)? ;

[GitHub] [spark] cloud-fan commented on a diff in pull request #37749: [SPARK-40295][SQL] Allow v2 functions with literal args in write distribution/ordering

cloud-fan commented on code in PR #37749: URL: https://github.com/apache/spark/pull/37749#discussion_r960163485 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala: ## @@ -105,18 +105,27 @@ object V2ExpressionUtils extends

[GitHub] [spark] ulysses-you commented on a diff in pull request #37612: [SPARK-39915][SQL] Ensure the output partitioning is user-specified in AQE

ulysses-you commented on code in PR #37612: URL: https://github.com/apache/spark/pull/37612#discussion_r960162375 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -2643,6 +2643,49 @@ class AdaptiveQueryExecSuite

[GitHub] [spark] cloud-fan closed pull request #37483: [SPARK-40112][SQL] Improve the TO_BINARY() function

cloud-fan closed pull request #37483: [SPARK-40112][SQL] Improve the TO_BINARY() function URL: https://github.com/apache/spark/pull/37483 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] cloud-fan commented on pull request #37483: [SPARK-40112][SQL] Improve the TO_BINARY() function

cloud-fan commented on PR #37483: URL: https://github.com/apache/spark/pull/37483#issuecomment-1233655860 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] cloud-fan commented on a diff in pull request #37612: [SPARK-39915][SQL] Ensure the output partitioning is user-specified in AQE

cloud-fan commented on code in PR #37612: URL: https://github.com/apache/spark/pull/37612#discussion_r960158919 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -2643,6 +2643,49 @@ class AdaptiveQueryExecSuite

[GitHub] [spark] cloud-fan commented on a diff in pull request #37612: [SPARK-39915][SQL] Ensure the output partitioning is user-specified in AQE

cloud-fan commented on code in PR #37612: URL: https://github.com/apache/spark/pull/37612#discussion_r960157740 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelation.scala: ## @@ -130,7 +130,8 @@ abstract class PropagateEmptyRelationBase

[GitHub] [spark] ulysses-you commented on a diff in pull request #37612: [SPARK-39915][SQL] Ensure the output partitioning is user-specified in AQE

ulysses-you commented on code in PR #37612: URL: https://github.com/apache/spark/pull/37612#discussion_r960155139 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala: ## @@ -40,7 +40,15 @@ case class LogicalQueryStage( override def

[GitHub] [spark] amaliujia commented on a diff in pull request #37750: [SPARK-40296] Error class for DISTINCT function not found

amaliujia commented on code in PR #37750: URL: https://github.com/apache/spark/pull/37750#discussion_r960150069 ## sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala: ## @@ -4473,6 +4473,16 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with

[GitHub] [spark] amaliujia commented on a diff in pull request #37750: [SPARK-40296] Error class for DISTINCT function not found

amaliujia commented on code in PR #37750: URL: https://github.com/apache/spark/pull/37750#discussion_r960149344 ## sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala: ## @@ -4473,6 +4473,16 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with

[GitHub] [spark] amaliujia commented on a diff in pull request #37750: [SPARK-40296] Error class for DISTINCT function not found

amaliujia commented on code in PR #37750: URL: https://github.com/apache/spark/pull/37750#discussion_r960149145 ## sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala: ## @@ -4473,6 +4473,16 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with

[GitHub] [spark] amaliujia commented on pull request #37750: [SPARK-40296] Error class for DISTINCT function not found

amaliujia commented on PR #37750: URL: https://github.com/apache/spark/pull/37750#issuecomment-1233633396 R: @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] amaliujia opened a new pull request, #37750: [SPARK-40296] Error class for DISTINCT function not found

amaliujia opened a new pull request, #37750: URL: https://github.com/apache/spark/pull/37750 ### What changes were proposed in this pull request? When users have a query like `SELECT SUM(a), DISTINCT(a, b) FROM table`, the DISTINCT is treated as a function but cannot be

[GitHub] [spark] ulysses-you commented on pull request #37463: [SPARK-40033][SQL] Nested schema pruning support through element_at

ulysses-you commented on PR #37463: URL: https://github.com/apache/spark/pull/37463#issuecomment-1233630298 thank you @viirya , rebased -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] zhengruifeng commented on a diff in pull request #37728: [SPARK-40276][CORE] Reduce the result size of RDD.takeOrdered

zhengruifeng commented on code in PR #37728: URL: https://github.com/apache/spark/pull/37728#discussion_r960145512 ## core/src/main/scala/org/apache/spark/rdd/RDD.scala: ## @@ -1523,22 +1523,21 @@ abstract class RDD[T: ClassTag]( * @return an array of top elements */

[GitHub] [spark] zhengruifeng commented on pull request #37728: [SPARK-40276][CORE] Reduce the result size of RDD.takeOrdered

zhengruifeng commented on PR #37728: URL: https://github.com/apache/spark/pull/37728#issuecomment-1233622631 Merged into master, thank you all! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] zhengruifeng closed pull request #37728: [SPARK-40276][CORE] Reduce the result size of RDD.takeOrdered

zhengruifeng closed pull request #37728: [SPARK-40276][CORE] Reduce the result size of RDD.takeOrdered URL: https://github.com/apache/spark/pull/37728 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] itholic commented on pull request #37210: add ignore for the recently added and failing mypy error 'type-var' SPARK-39811

itholic commented on PR #37210: URL: https://github.com/apache/spark/pull/37210#issuecomment-1233618886 Could you check the "Workflow run detection failed" in the https://github.com/apache/spark/pull/37210/checks?check_run_id=7404721464 ? You should enable the Github Actions workflow

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960139158 ## project/MimaExcludes.scala: ## @@ -124,63 +170,12 @@ object MimaExcludes { // Avro source implementation is internal.

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960138266 ## project/MimaExcludes.scala: ## @@ -124,63 +170,12 @@ object MimaExcludes { // Avro source implementation is internal.

[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on code in PR #37741: URL: https://github.com/apache/spark/pull/37741#discussion_r960136975 ## project/MimaExcludes.scala: ## @@ -124,63 +170,12 @@ object MimaExcludes { // Avro source implementation is internal.

[GitHub] [spark] LuciferYang commented on pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on PR #37741: URL: https://github.com/apache/spark/pull/37741#issuecomment-1233598133 > > Maybe, do you think you can clean up more like [SPARK-36004](https://issues.apache.org/jira/browse/SPARK-36004)? > > like

[GitHub] [spark] JoshRosen closed pull request #37713: [SPARK-40261][CORE]Exclude DirectTaskResult metadata when calculating result size

JoshRosen closed pull request #37713: [SPARK-40261][CORE]Exclude DirectTaskResult metadata when calculating result size URL: https://github.com/apache/spark/pull/37713 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] LuciferYang commented on pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0

LuciferYang commented on PR #37741: URL: https://github.com/apache/spark/pull/37741#issuecomment-1233590169 > f2ed5d8 like [f2ed5d8](https://github.com/apache/spark/pull/37741/commits/f2ed5d8118062bb382a1bf8631db3fedad5dfc92)? -- This is an automated message from the Apache Git

[GitHub] [spark] github-actions[bot] closed pull request #36606: [SPARK-39232][CORE] History Server Main Page App List Filtering

github-actions[bot] closed pull request #36606: [SPARK-39232][CORE] History Server Main Page App List Filtering URL: https://github.com/apache/spark/pull/36606 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] vitaliili-db commented on pull request #37621: [SPARK-40185][SQL] Remove column suggestion when the candidate list is empty

vitaliili-db commented on PR #37621: URL: https://github.com/apache/spark/pull/37621#issuecomment-1233579923 @MaxGekk friendly ping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] viirya commented on pull request #37463: [SPARK-40033][SQL] Nested schema pruning support through element_at

viirya commented on PR #37463: URL: https://github.com/apache/spark/pull/37463#issuecomment-1233564621 @ulysses-you Could you retrigger the CI? Since last CI run is many days ago, it is safer to run it again before merging. -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] sigmod commented on a diff in pull request #37697: [SPARK-40248][SQL] Use larger number of bits to build Bloom filter

sigmod commented on code in PR #37697: URL: https://github.com/apache/spark/pull/37697#discussion_r960091832 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/BloomFilterAggregate.scala: ## @@ -55,6 +55,13 @@ case class BloomFilterAggregate(

[GitHub] [spark] xkrogen commented on a diff in pull request #36506: [SPARK-25050][SQL] Avro: writing complex unions

xkrogen commented on code in PR #36506: URL: https://github.com/apache/spark/pull/36506#discussion_r960091825 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala: ## @@ -218,6 +218,17 @@ private[sql] class AvroSerializer( val numFields =

[GitHub] [spark] sigmod commented on a diff in pull request #37697: [SPARK-40248][SQL] Use larger number of bits to build Bloom filter

sigmod commented on code in PR #37697: URL: https://github.com/apache/spark/pull/37697#discussion_r960030651 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/BloomFilterAggregate.scala: ## @@ -55,6 +55,13 @@ case class BloomFilterAggregate(

[GitHub] [spark] xkrogen commented on pull request #37634: [SPARK-40199][SQL] Provide useful error when projecting a non-null column encounters null value

xkrogen commented on PR #37634: URL: https://github.com/apache/spark/pull/37634#issuecomment-1233486676 Pushed up new commits rebasing on latest changes. cc @cloud-fan @dongjoon-hyun in case either of you are interested. -- This is an automated message from the Apache Git Service.

[GitHub] [spark] aokolnychyi commented on a diff in pull request #37749: [SPARK-40295][SQL] Allow v2 functions with literal args in write distribution/ordering

aokolnychyi commented on code in PR #37749: URL: https://github.com/apache/spark/pull/37749#discussion_r960072719 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala: ## @@ -105,18 +105,27 @@ object V2ExpressionUtils extends

[GitHub] [spark] xkrogen commented on a diff in pull request #37634: [SPARK-40199][SQL] Provide useful error when projecting a non-null column encounters null value

xkrogen commented on code in PR #37634: URL: https://github.com/apache/spark/pull/37634#discussion_r960070035 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala: ## @@ -252,28 +266,44 @@ object

[GitHub] [spark] huaxingao commented on a diff in pull request #37746: [SPARK-40293][SQL] Make the V2 table error message more meaningful

huaxingao commented on code in PR #37746: URL: https://github.com/apache/spark/pull/37746#discussion_r960066037 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -844,10 +846,6 @@ private[sql] object QueryCompilationErrors extends

[GitHub] [spark] aokolnychyi commented on a diff in pull request #37749: [SPARK-40295][SQL] Allow v2 functions with literal args in write distribution/ordering

aokolnychyi commented on code in PR #37749: URL: https://github.com/apache/spark/pull/37749#discussion_r960057109 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala: ## @@ -105,18 +105,27 @@ object V2ExpressionUtils extends

[GitHub] [spark] aokolnychyi commented on pull request #37749: [SPARK-40295][SQL] Allow v2 functions with literal args in write distribution/ordering

aokolnychyi commented on PR #37749: URL: https://github.com/apache/spark/pull/37749#issuecomment-1233450883 @cloud-fan @sunchao @pan3793, could you take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] aokolnychyi opened a new pull request, #37749: [SPARK-40295][SQL] Allow v2 functions with literal args in write distribution/ordering

aokolnychyi opened a new pull request, #37749: URL: https://github.com/apache/spark/pull/37749 ### What changes were proposed in this pull request? This PR adapts `V2ExpressionUtils` to support arbitrary transforms with multiple args that are either references or

[GitHub] [spark] amaliujia commented on a diff in pull request #37746: [SPARK-40293][SQL] Make the V2 table error message more meaningful

amaliujia commented on code in PR #37746: URL: https://github.com/apache/spark/pull/37746#discussion_r960036859 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -844,10 +846,6 @@ private[sql] object QueryCompilationErrors extends

[GitHub] [spark] khalidmammadov commented on pull request #37748: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call

khalidmammadov commented on PR #37748: URL: https://github.com/apache/spark/pull/37748#issuecomment-1233379236 This is second solution to the issue. 1st was https://github.com/apache/spark/pull/37650 -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] khalidmammadov commented on pull request #37650: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call

khalidmammadov commented on PR #37650: URL: https://github.com/apache/spark/pull/37650#issuecomment-1233378272 follow up PR: https://github.com/apache/spark/pull/37748 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] khalidmammadov opened a new pull request, #37748: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call

khalidmammadov opened a new pull request, #37748: URL: https://github.com/apache/spark/pull/37748 ### What changes were proposed in this pull request? PySpark atan2, hypot, pow and pmod functions marked as accepting float type as argument but produce error when called together. For

[GitHub] [spark] dongjoon-hyun commented on pull request #37745: [SPARK-33605][BUILD] Add `gcs-connector` to `hadoop-cloud` module

dongjoon-hyun commented on PR #37745: URL: https://github.com/apache/spark/pull/37745#issuecomment-1233372943 - Yes, only for better GCS support for the users who use `-Phadoop-cloud`. - Apache Spark distribution doesn't use `-Phadoop-cloud` during our release process. So, the publish

[GitHub] [spark] EnricoMi commented on a diff in pull request #37407: [SPARK-39876][SQL] Add UNPIVOT to SQL syntax

EnricoMi commented on code in PR #37407: URL: https://github.com/apache/spark/pull/37407#discussion_r959969544 ## docs/sql-ref-syntax-qry-select.md: ## @@ -75,7 +76,7 @@ SELECT [ hints , ... ] [ ALL | DISTINCT ] { [ [ named_expression | regex_column_ An expression with

  1   2   >