Re: [PR] [SPARK-48358][SQL] Support for REPEAT statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47756: URL: https://github.com/apache/spark/pull/47756#discussion_r1738866375 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala: ## @@ -323,3 +323,58 @@ class WhileStatementExec( body.reset() } }

Re: [PR] [SPARK-48700] [SQL] Mode expression for complex types (all collations) [spark]

2024-08-30 Thread via GitHub
MaxGekk commented on code in PR #47154: URL: https://github.com/apache/spark/pull/47154#discussion_r1738971754 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Mode.scala: ## @@ -50,17 +50,19 @@ case class Mode( override def inputTypes: Seq[A

Re: [PR] [SPARK-49459][CORE][SHUFFLE] Support `CRC32C` for Shuffle Checksum [spark]

2024-08-30 Thread via GitHub
dongjoon-hyun closed pull request #47929: [SPARK-49459][CORE][SHUFFLE] Support `CRC32C` for Shuffle Checksum URL: https://github.com/apache/spark/pull/47929 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-49466][BUILD] Upgrade `zstd-jni` to 1.5.6-5 [spark]

2024-08-30 Thread via GitHub
dongjoon-hyun closed pull request #47934: [SPARK-49466][BUILD] Upgrade `zstd-jni` to 1.5.6-5 URL: https://github.com/apache/spark/pull/47934 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Handle Concurrent Create Table AS Select FROM commands [spark]

2024-08-30 Thread via GitHub
konjac commented on code in PR #47820: URL: https://github.com/apache/spark/pull/47820#discussion_r1739152667 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala: ## @@ -55,6 +57,19 @@ case class WriteToDataSourceV2( copy(q

Re: [PR] [SPARK-48682][SQL][FOLLOW-UP] Changed initCap behaviour with UTF8_BINARY collation [spark]

2024-08-30 Thread via GitHub
MaxGekk commented on code in PR #47771: URL: https://github.com/apache/spark/pull/47771#discussion_r1739193793 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java: ## @@ -550,6 +549,152 @@ public static UTF8String toTitleCase(final UTF

Re: [PR] [SPARK-49411][SS] Communicate CheckpointID between driver and stateful operators [spark]

2024-08-30 Thread via GitHub
WweiL commented on code in PR #47895: URL: https://github.com/apache/spark/pull/47895#discussion_r1739209741 ## sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreIntegrationSuite.scala: ## @@ -222,6 +371,122 @@ class RocksDBStateStoreIntegra

Re: [PR] [SPARK-49443][SQL][PYTHON] Implement to_variant_object expression and make schema_of_variant expressions print OBJECT for for Variant Objects [spark]

2024-08-30 Thread via GitHub
chenhao-db commented on code in PR #47907: URL: https://github.com/apache/spark/pull/47907#discussion_r1739215575 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala: ## @@ -114,6 +114,81 @@ case class IsVariantNull(child: Ex

[PR] [WIP][SPARK-49474][SS] Classify Error class for FlatMapGroupsWithState user function error [spark]

2024-08-30 Thread via GitHub
liviazhu-db opened a new pull request, #47940: URL: https://github.com/apache/spark/pull/47940 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

Re: [PR] [SPARK-48355][SQL] Support for CASE statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47672: URL: https://github.com/apache/spark/pull/47672#discussion_r1739366597 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/SqlScriptingParserSuite.scala: ## @@ -677,4 +959,17 @@ class SqlScriptingParserSuite extends Spark

Re: [PR] [SPARK-48355][SQL] Support for CASE statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47672: URL: https://github.com/apache/spark/pull/47672#discussion_r1739367047 ## sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4: ## @@ -83,6 +84,13 @@ ifElseStatement (ELSE elseBody=compoundBody)? END IF

Re: [PR] [SPARK-48355][SQL] Support for CASE statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47672: URL: https://github.com/apache/spark/pull/47672#discussion_r1739368128 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -247,6 +247,33 @@ class AstBuilder extends DataTypeAstBuilder WhileSt

Re: [PR] [SPARK-48355][SQL] Support for CASE statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47672: URL: https://github.com/apache/spark/pull/47672#discussion_r1739369364 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -247,6 +247,33 @@ class AstBuilder extends DataTypeAstBuilder WhileSt

[PR] [SPARK-49476] Fix nullability of base64 function [spark]

2024-08-30 Thread via GitHub
Kimahriman opened a new pull request, #47941: URL: https://github.com/apache/spark/pull/47941 ### What changes were proposed in this pull request? Fix the nullability of the `Base64` expression to be based on the child's nullability, and not always be nullable. ### Why

Re: [PR] [SPARK-48355][SQL] Support for CASE statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47672: URL: https://github.com/apache/spark/pull/47672#discussion_r1739380661 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlScriptingLogicalOperators.scala: ## @@ -89,3 +89,18 @@ case class WhileStatement( conditio

Re: [PR] [SPARK-48355][SQL] Support for CASE statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47672: URL: https://github.com/apache/spark/pull/47672#discussion_r1739380661 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlScriptingLogicalOperators.scala: ## @@ -89,3 +89,18 @@ case class WhileStatement( conditio

Re: [PR] [SPARK-49476] Fix nullability of base64 function [spark]

2024-08-30 Thread via GitHub
Kimahriman commented on PR #47941: URL: https://github.com/apache/spark/pull/47941#issuecomment-2322290888 @dongjoon-hyun @yaooqinn @wForget -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] [SPARK-48355][SQL] Support for CASE statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47672: URL: https://github.com/apache/spark/pull/47672#discussion_r1739397808 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala: ## @@ -323,3 +323,78 @@ class WhileStatementExec( body.reset() } }

Re: [PR] [SPARK-48355][SQL] Support for CASE statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47672: URL: https://github.com/apache/spark/pull/47672#discussion_r1739399723 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreter.scala: ## @@ -92,6 +92,16 @@ case class SqlScriptingInterpreter() { trans

Re: [PR] [SPARK-48355][SQL] Support for CASE statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47672: URL: https://github.com/apache/spark/pull/47672#discussion_r1739402808 ## sql/core/src/test/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreterSuite.scala: ## @@ -368,6 +368,238 @@ class SqlScriptingInterpreterSuite extends Quer

Re: [PR] [SPARK-48355][SQL] Support for CASE statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47672: URL: https://github.com/apache/spark/pull/47672#discussion_r1739402808 ## sql/core/src/test/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreterSuite.scala: ## @@ -368,6 +368,238 @@ class SqlScriptingInterpreterSuite extends Quer

Re: [PR] [SPARK-48355][SQL] Support for CASE statement [spark]

2024-08-30 Thread via GitHub
davidm-db commented on code in PR #47672: URL: https://github.com/apache/spark/pull/47672#discussion_r1739403703 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -229,6 +229,31 @@ class AstBuilder extends DataTypeAstBuilder ) }

Re: [PR] [SPARK-49016][SQL] Restore the behavior that queries from raw CSV files are disallowed when only include corrupt record column and assign name to `_LEGACY_ERROR_TEMP_1285` [spark]

2024-09-01 Thread via GitHub
cloud-fan commented on code in PR #47506: URL: https://github.com/apache/spark/pull/47506#discussion_r1740348480 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala: ## @@ -1781,6 +1781,32 @@ abstract class CSVSuite } } + test("SPA

Re: [PR] [SPARK-49085][SQL] Remove special casing for Protobuf functions in Connect [spark]

2024-09-01 Thread via GitHub
zhengruifeng commented on code in PR #47885: URL: https://github.com/apache/spark/pull/47885#discussion_r1740290229 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala: ## @@ -919,6 +919,8 @@ object FunctionRegistry { registerInternalE

Re: [PR] [SPARK-49044][SQL] ValidateExternalType should return child in error [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on code in PR #47522: URL: https://github.com/apache/spark/pull/47522#discussion_r1740184797 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -2177,6 +2177,12 @@ ], "sqlState" : "42001" }, + "INVALID_EXTERNAL_TYPE" : { +"m

Re: [PR] [SPARK-46395][CORE] Assign Spark configs to groups for use in documentation [spark]

2024-08-30 Thread via GitHub
github-actions[bot] closed pull request #44755: [SPARK-46395][CORE] Assign Spark configs to groups for use in documentation URL: https://github.com/apache/spark/pull/44755 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] [SPARK-49459][CORE][SHUFFLE] Support `CRC32C` for Shuffle Checksum [spark]

2024-09-01 Thread via GitHub
yaooqinn commented on PR #47929: URL: https://github.com/apache/spark/pull/47929#issuecomment-2323851897 Thank you @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] [WIP][SQL] Handle invalid UTF-8 strings in StringLocate [spark]

2024-08-31 Thread via GitHub
dongjoon-hyun closed pull request #47694: [WIP][SQL] Handle invalid UTF-8 strings in StringLocate URL: https://github.com/apache/spark/pull/47694 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-49460][SQL] Remove `cleanupResource()` from EmptyRelationExec [spark]

2024-08-30 Thread via GitHub
LuciferYang commented on PR #47931: URL: https://github.com/apache/spark/pull/47931#issuecomment-2322729677 Merged into master. Thanks @liuzqt -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-46698][SPARK-46895][CORE][3.5] Replace Timer with single thread scheduled executor [spark]

2024-09-01 Thread via GitHub
yaooqinn commented on PR #47945: URL: https://github.com/apache/spark/pull/47945#issuecomment-2323722626 @jshmchenxi, Can we submit separate pull requests along with their primitive forms? -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] [SPARK-49455][SQL][TESTS] Refactor `StagingInMemoryTableCatalog` to override the non-deprecated functions [spark]

2024-09-01 Thread via GitHub
LuciferYang closed pull request #47924: [SPARK-49455][SQL][TESTS] Refactor `StagingInMemoryTableCatalog` to override the non-deprecated functions URL: https://github.com/apache/spark/pull/47924 -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] [TEST][WIP][][SQL] Introduction of error handling mechanism in SQL Scripting [spark]

2024-08-31 Thread via GitHub
dongjoon-hyun commented on PR #47356: URL: https://github.com/apache/spark/pull/47356#issuecomment-2323021314 Hi, @miland-db . You may want to open a PR to your forked repository . It will give you the GitHub Action test results in the same way. Both uses your GitHub Action resources in any

Re: [PR] [SPARK-48549][SQL] Restrict the number of parameters for function `sentences` to `1` or `3` [spark]

2024-09-01 Thread via GitHub
panbingkun commented on PR #46880: URL: https://github.com/apache/spark/pull/46880#issuecomment-2323862431 > We **shall not break** existing SQL user's code only because it is inconsistent to Scala/Python APIs. As I said let's document the existing behaviour of SQL func first of all, and se

Re: [PR] [SPARK-49455][SQL][TESTS] Refactor `StagingInMemoryTableCatalog` to override the non-deprecated functions [spark]

2024-09-01 Thread via GitHub
LuciferYang commented on PR #47924: URL: https://github.com/apache/spark/pull/47924#issuecomment-2323862483 Merged into master. Thanks @yaooqinn again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-49453] Add quotes for number [spark-kubernetes-operator]

2024-08-31 Thread via GitHub
dongjoon-hyun closed pull request #112: [SPARK-49453] Add quotes for number URL: https://github.com/apache/spark-kubernetes-operator/pull/112 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [SPARK-49480][CORE] Fix NullPointerException from `SparkThrowableHelper.isInternalError` [spark]

2024-09-01 Thread via GitHub
jshmchenxi commented on PR #47946: URL: https://github.com/apache/spark/pull/47946#issuecomment-2323702153 > After your changes, this check `st.getErrorClass != null` is redundant. > > https://github.com/apache/spark/blob/c58148da5496245403b55c3fc423d35f3a669c79/core/src/main/scala/or

Re: [PR] [SPARK-48354][SQL] JDBC Connectors predicate pushdown testing [spark]

2024-08-30 Thread via GitHub
github-actions[bot] commented on PR #46642: URL: https://github.com/apache/spark/pull/46642#issuecomment-2322629511 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-49476] Fix nullability of base64 function [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on code in PR #47941: URL: https://github.com/apache/spark/pull/47941#discussion_r1739994194 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala: ## @@ -467,6 +467,12 @@ class StringExpressionsSuite extends Spar

Re: [PR] [SPARK-49480][CORE] Fix NullPointerException from `SparkThrowableHelper.isInternalError` [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on PR #47946: URL: https://github.com/apache/spark/pull/47946#issuecomment-2323863986 > Thanks for review! Removing it will change the condition from false to true when st.getErrorClass is null, which doesn't seem to be right. I see, it checks not only internal but n

Re: [PR] [SPARK-49483][BUILD] Upgrade `commons-lang3` to 3.17.0 [spark]

2024-09-01 Thread via GitHub
LuciferYang closed pull request #47948: [SPARK-49483][BUILD] Upgrade `commons-lang3` to 3.17.0 URL: https://github.com/apache/spark/pull/47948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[PR] [WIP][SPARK-49488][SQL] Improve the DS V2 pushdown framework for DayOfWeek and WeekDay [spark]

2024-09-01 Thread via GitHub
beliefer opened a new pull request, #47951: URL: https://github.com/apache/spark/pull/47951 ### What changes were proposed in this pull request? This PR propose to improve the DS V2 pushdown framework for `DayOfWeek` and `WeekDay`. ### Why are the changes needed? 1. The

Re: [PR] [SPARK-49483][BUILD] Upgrade `commons-lang3` to 3.17.0 [spark]

2024-09-01 Thread via GitHub
LuciferYang commented on PR #47948: URL: https://github.com/apache/spark/pull/47948#issuecomment-2323865059 Merged into master. Thanks @panbingkun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-41469][CORE] Avoid unnecessary task rerun on decommissioned executor lost if shuffle data migrated [spark]

2024-09-01 Thread via GitHub
Ngone51 commented on code in PR #39011: URL: https://github.com/apache/spark/pull/39011#discussion_r1740261348 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1046,17 +1048,45 @@ private[spark] class TaskSetManager( /** Called by TaskScheduler

Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]

2024-08-31 Thread via GitHub
gengliangwang commented on PR #47500: URL: https://github.com/apache/spark/pull/47500#issuecomment-2323163355 Thanks, merging to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [SPARK-49275][SQL] Fix return type nullness of the xpath expression [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on PR #47796: URL: https://github.com/apache/spark/pull/47796#issuecomment-2323236624 @chenhao-db Do the previous Spark versions suffer from the issue too like `branch-3.5`? -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] [SPARK-46798][Structured Streaming] Kafka custom partition location assignment (rack awareness) [spark]

2024-08-31 Thread via GitHub
neilramaswamy commented on PR #46863: URL: https://github.com/apache/spark/pull/46863#issuecomment-2323192496 @rschwagercharter, can you enable public comment-access on that doc? Or, at least, my email: n...@ramaswamy.org. Thanks. -- This is an automated message from the Apache Git Servic

Re: [PR] [SPARK-49044][SQL] ValidateExternalType should return child in error [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on code in PR #47522: URL: https://github.com/apache/spark/pull/47522#discussion_r1740183686 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala: ## @@ -556,6 +556,43 @@ class ObjectExpressionsSuite extends Spar

Re: [PR] [SPARK-49476][SQL] Fix nullability of base64 function [spark]

2024-09-01 Thread via GitHub
LuciferYang commented on code in PR #47941: URL: https://github.com/apache/spark/pull/47941#discussion_r1740358827 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala: ## @@ -467,6 +467,14 @@ class StringExpressionsSuite extends

Re: [PR] [SPARK-49468] Add Hot Properties Reload Test [spark-kubernetes-operator]

2024-08-31 Thread via GitHub
dongjoon-hyun closed pull request #115: [SPARK-49468] Add Hot Properties Reload Test URL: https://github.com/apache/spark-kubernetes-operator/pull/115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-49480][CORE] Fix NullPointerException from `SparkThrowableHelper.isInternalError` [spark]

2024-09-01 Thread via GitHub
cloud-fan commented on PR #47946: URL: https://github.com/apache/spark/pull/47946#issuecomment-2323874036 thanks, merging to master/3.5! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] [SPARK-47336][SQL][CONNECT] Provide to PySpark a functionality to get estimated size of DataFrame in bytes [spark]

2024-09-01 Thread via GitHub
SemyonSinchenko commented on PR #46368: URL: https://github.com/apache/spark/pull/46368#issuecomment-2323232116 @HyukjinKwon @zhengruifeng @cloud-fan Sorry for tagging but maybe you can take a look again? I fixed everything from the last review round... Tnx in advance! -- This is an auto

Re: [PR] [SPARK-48309][YARN]Stop am retry, in situations where some errors and retries may not be successful [spark]

2024-09-01 Thread via GitHub
github-actions[bot] commented on PR #46620: URL: https://github.com/apache/spark/pull/46620#issuecomment-2323563725 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-49383][SQL][PYTHON][CONNECT] Support Transpose DataFrame API [spark]

2024-09-01 Thread via GitHub
xinrong-meng commented on PR #47884: URL: https://github.com/apache/spark/pull/47884#issuecomment-2323642874 I verified that `transpose` works on Spark Connect Scala Client as ``` scala> spark.range(2) res0: org.apache.spark.sql.Dataset[java.lang.Long] = [id: bigint] scala> s

Re: [PR] [SPARK-49480][CORE] Fix NullPointerException from `SparkThrowableHelper.isInternalError` [spark]

2024-09-01 Thread via GitHub
cloud-fan closed pull request #47946: [SPARK-49480][CORE] Fix NullPointerException from `SparkThrowableHelper.isInternalError` URL: https://github.com/apache/spark/pull/47946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] [SPARK-46698][SPARK-46895][CORE][3.5] Replace Timer with single thread scheduled executor [spark]

2024-09-01 Thread via GitHub
LuciferYang commented on code in PR #47945: URL: https://github.com/apache/spark/pull/47945#discussion_r1740282052 ## core/src/main/scala/org/apache/spark/BarrierCoordinator.scala: ## @@ -51,7 +52,8 @@ private[spark] class BarrierCoordinator( // TODO SPARK-25030 Create a Ti

[PR] [SPARK-49482][SQL] Refactor V2 parquet datasource [spark]

2024-08-31 Thread via GitHub
mikecsx opened a new pull request, #47947: URL: https://github.com/apache/spark/pull/47947 ### What changes were proposed in this pull request? Split V2 parquet datasource case classes into pairs of abstract classes and case classes ### Why are the changes needed? To

Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]

2024-08-31 Thread via GitHub
gengliangwang closed pull request #47500: [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` URL: https://github.com/apache/spark/pull/47500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [SPARK-49476][SQL] Fix nullability of base64 function [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on code in PR #47941: URL: https://github.com/apache/spark/pull/47941#discussion_r1740365156 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala: ## @@ -467,6 +467,14 @@ class StringExpressionsSuite extends Spar

Re: [PR] [WIP][SQL] Handle invalid UTF-8 strings in StringInstr [spark]

2024-08-31 Thread via GitHub
dongjoon-hyun closed pull request #47695: [WIP][SQL] Handle invalid UTF-8 strings in StringInstr URL: https://github.com/apache/spark/pull/47695 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] [SPARK-49476][SQL] Fix nullability of base64 function [spark]

2024-09-01 Thread via GitHub
MaxGekk closed pull request #47941: [SPARK-49476][SQL] Fix nullability of base64 function URL: https://github.com/apache/spark/pull/47941 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] [SPARK-49485] Fix speculative task hang bug due to remaining executor… [spark]

2024-09-01 Thread via GitHub
buska88 opened a new pull request, #47949: URL: https://github.com/apache/spark/pull/47949 … lay on same host ### What changes were proposed in this pull request? Modify ExecutorAllocationManager calculates maxNeeded logic.If remaining executors on a same host and pendi

Re: [PR] [SPARK-48298][Core] Add TCP mode to StatsD sink [spark]

2024-08-31 Thread via GitHub
github-actions[bot] commented on PR #46604: URL: https://github.com/apache/spark/pull/46604#issuecomment-2323086424 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] add possibility to set log filename & disable spark log rotation [spark]

2024-08-31 Thread via GitHub
dongjoon-hyun commented on PR #47373: URL: https://github.com/apache/spark/pull/47373#issuecomment-2323022000 Let me close this based on the Apache Spark community backporting policy which allows only bug fixes. For the PR on master branch, let's continue the discussion, @Tocard . -

Re: [PR] [SPARK-49275][SQL] Fix return type nullness of the xpath expression [spark]

2024-09-01 Thread via GitHub
chenhao-db commented on PR #47796: URL: https://github.com/apache/spark/pull/47796#issuecomment-2323717201 @MaxGekk Yes, I believe this issue has existed since `xpath` was introduced. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] [SPARK-49480][CORE] Fix NullPointerException from `SparkThrowableHelper.isInternalError` [spark]

2024-08-31 Thread via GitHub
jshmchenxi commented on PR #47946: URL: https://github.com/apache/spark/pull/47946#issuecomment-2322887799 Kindly ping @bozhang2820 @cloud-fan. Please take a look, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[PR] [SPARK-46698][SPARK-46895][CORE][3.5] Replace Timer with single thread scheduled executor [spark]

2024-08-30 Thread via GitHub
jshmchenxi opened a new pull request, #47945: URL: https://github.com/apache/spark/pull/47945 backport #44701 and #44718 to 3.5 branch as a fix for [SPARK-49479](https://issues.apache.org/jira/browse/SPARK-49479): Non-daemon Timer prevents Spark driver JVM from stopping. ### What

Re: [PR] [SPARK-49460][SQL] Remove `cleanupResource()` from EmptyRelationExec [spark]

2024-08-30 Thread via GitHub
LuciferYang commented on PR #47931: URL: https://github.com/apache/spark/pull/47931#issuecomment-2322731331 @liuzqt I would like to confirm, does this issue really exist in 3.5.x? It seems that `EmptyRelationExec` does not exist in branch-3.5. -- This is an automated message from the Apac

Re: [PR] [SPARK-46698][SPARK-46895][CORE][3.5] Replace Timer with single thread scheduled executor [spark]

2024-08-31 Thread via GitHub
jshmchenxi commented on PR #47945: URL: https://github.com/apache/spark/pull/47945#issuecomment-2322885017 > This is only a minor improvement for Spark 4.0, not a bug fix, and it should not be backported to 3.x. @LuciferYang Thanks for reply! It actually includes a fix for https://is

Re: [PR] [SPARK-48549][SQL] Restrict the number of parameters for function `sentences` to `1` or `3` [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on PR #46880: URL: https://github.com/apache/spark/pull/46880#issuecomment-2323250528 We **shall not break** existing SQL user's code only because it is inconsistent to Scala/Python APIs. As I said let's document the existing behaviour of SQL func first of all, and separat

Re: [PR] [SPARK-49044][SQL] ValidateExternalType should return child in error [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on code in PR #47522: URL: https://github.com/apache/spark/pull/47522#discussion_r1740183995 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala: ## @@ -2057,14 +2055,12 @@ case class ValidateExternalType(child: Expres

Re: [PR] [SPARK-49275][SQL] Fix return type nullness of the xpath expression [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on PR #47796: URL: https://github.com/apache/spark/pull/47796#issuecomment-2323899036 +1, LGTM. Merging to master/3.5. Thank you, @chenhao-db and @HyukjinKwon for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] [Do not review] Fix group by on collated string [spark]

2024-08-31 Thread via GitHub
dongjoon-hyun closed pull request #47520: [Do not review] Fix group by on collated string URL: https://github.com/apache/spark/pull/47520 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-49275][SQL] Fix return type nullness of the xpath expression [spark]

2024-09-01 Thread via GitHub
MaxGekk closed pull request #47796: [SPARK-49275][SQL] Fix return type nullness of the xpath expression URL: https://github.com/apache/spark/pull/47796 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] [SPARK-49460][SQL] Remove `cleanupResource()` from EmptyRelationExec [spark]

2024-08-30 Thread via GitHub
liuzqt commented on PR #47931: URL: https://github.com/apache/spark/pull/47931#issuecomment-2322639747 @LuciferYang Updated. Could you pls help merge this PR? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] add possibility to set log filename & disable spark log rotation [spark]

2024-08-31 Thread via GitHub
dongjoon-hyun closed pull request #47373: add possibility to set log filename & disable spark log rotation URL: https://github.com/apache/spark/pull/47373 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] [SPARK-49480][CORE] Fix NullPointerException from `SparkThrowableHelper.isInternalError` [spark]

2024-09-01 Thread via GitHub
jshmchenxi commented on PR #47946: URL: https://github.com/apache/spark/pull/47946#issuecomment-2323900490 @cloud-fan Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] [WIP][SQL] Handle invalid UTF-8 strings in StringTrim [spark]

2024-08-31 Thread via GitHub
dongjoon-hyun closed pull request #47696: [WIP][SQL] Handle invalid UTF-8 strings in StringTrim URL: https://github.com/apache/spark/pull/47696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] [SPARK-49275][SQL] Fix return type nullness of the xpath expression [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on PR #47796: URL: https://github.com/apache/spark/pull/47796#issuecomment-2323903843 @chenhao-db The changes cause conflicts in `branch-3.5`. Please, open a separate PR with backport. -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] [SPARK-49044][SQL] ValidateExternalType should return child in error [spark]

2024-09-01 Thread via GitHub
mrk-andreev commented on code in PR #47522: URL: https://github.com/apache/spark/pull/47522#discussion_r1740174819 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala: ## @@ -556,6 +556,43 @@ class ObjectExpressionsSuite extends

Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]

2024-08-31 Thread via GitHub
gengliangwang commented on PR #47500: URL: https://github.com/apache/spark/pull/47500#issuecomment-2323162517 @LuciferYang Thanks a lot for the investigation! I verified that your assumption is true ``` > InetAddress.getLocalHost val res0: java.net.InetAddress = hostname/192.1

Re: [PR] [SPARK-47263][SQL] Rename the error class _LEGACY_ERROR_TEMP_13[44-46… [spark]

2024-09-01 Thread via GitHub
PaysonXu commented on code in PR #46320: URL: https://github.com/apache/spark/pull/46320#discussion_r1740087167 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -3548,17 +3548,14 @@ private[sql] object QueryCompilationErrors extends

Re: [PR] [SPARK-46698][SPARK-46895][CORE][3.5] Replace Timer with single thread scheduled executor [spark]

2024-08-31 Thread via GitHub
LuciferYang commented on PR #47945: URL: https://github.com/apache/spark/pull/47945#issuecomment-2322881867 This is only a minor improvement for Spark 4.0, not a bug fix, and it should not be backported to 3.x. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [WIP][SQL] Handle invalid UTF-8 strings in SubstringIndex [spark]

2024-08-31 Thread via GitHub
dongjoon-hyun commented on PR #47693: URL: https://github.com/apache/spark/pull/47693#issuecomment-2323019375 Given the 3-week idle time, I'd recommend @viktorluc-db to open a testing PR to your forked repository. Both opening a PR to Apache Spark repo and opening a PR to your forked

Re: [PR] [SPARK-49460][SQL] Remove `cleanupResource()` from EmptyRelationExec [spark]

2024-08-30 Thread via GitHub
liuzqt commented on PR #47931: URL: https://github.com/apache/spark/pull/47931#issuecomment-2322759236 @LuciferYang sorry my bad, it doesn't exist in 3.5, so merging into master should be enough. Thank you for the reivew! -- This is an automated message from the Apache Git Service. To res

Re: [PR] [SPARK-49451] Allow duplicate keys in parse_json. [spark]

2024-09-01 Thread via GitHub
cloud-fan closed pull request #47920: [SPARK-49451] Allow duplicate keys in parse_json. URL: https://github.com/apache/spark/pull/47920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-49451] Allow duplicate keys in parse_json. [spark]

2024-09-01 Thread via GitHub
cloud-fan commented on PR #47920: URL: https://github.com/apache/spark/pull/47920#issuecomment-2323908607 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [WIP] Implement Levenshtein distance for utf8_lcase collation [spark]

2024-08-31 Thread via GitHub
dongjoon-hyun closed pull request #47689: [WIP] Implement Levenshtein distance for utf8_lcase collation URL: https://github.com/apache/spark/pull/47689 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] [WIP][SQL] Handle invalid UTF-8 strings in StringTrim [spark]

2024-08-31 Thread via GitHub
dongjoon-hyun commented on PR #47696: URL: https://github.com/apache/spark/pull/47696#issuecomment-2323019774 Given the 3-week idle time, I'd recommend @uros-db to open a testing PR to your forked repository. Both opening a PR to Apache Spark repo and opening a PR to your forked repo

Re: [PR] [SPARK-48385][SQL] Migrate the jdbc driver of mariadb from `2.x` to `3.x` [spark]

2024-09-01 Thread via GitHub
github-actions[bot] commented on PR #46655: URL: https://github.com/apache/spark/pull/46655#issuecomment-2323563690 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-49392] Catch errors when failing to write to external data source [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on code in PR #47873: URL: https://github.com/apache/spark/pull/47873#discussion_r1740001014 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -3982,6 +3982,14 @@ private[sql] object QueryCompilationErrors extends Q

Re: [PR] [SPARK-49044][SQL] ValidateExternalType should return child in error [spark]

2024-09-01 Thread via GitHub
mrk-andreev commented on code in PR #47522: URL: https://github.com/apache/spark/pull/47522#discussion_r1740164266 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala: ## @@ -556,6 +556,43 @@ class ObjectExpressionsSuite extends

Re: [PR] [SPARK-49085][SQL] Remove special casing for Protobuf functions in Connect [spark]

2024-09-01 Thread via GitHub
itholic commented on code in PR #47885: URL: https://github.com/apache/spark/pull/47885#discussion_r1740277447 ## sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -80,7 +80,6 @@ import org.apache.spark.sql.execution.streami

Re: [PR] [SPARK-47730][K8S] Support `APP_ID` and `EXECUTOR_ID` placeholders in labels [spark]

2024-08-31 Thread via GitHub
jshmchenxi commented on PR #46149: URL: https://github.com/apache/spark/pull/46149#issuecomment-2322888715 Glad to join the community. Thank you! @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [WIP] Implement Levenshtein distance for utf8_lcase collation [spark]

2024-08-31 Thread via GitHub
dongjoon-hyun commented on PR #47689: URL: https://github.com/apache/spark/pull/47689#issuecomment-2323019266 Given the 3-week idle time, I'd recommend @viktorluc-db to open a testing PR to your forked repository. Both opening a PR to Apache Spark repo and opening a PR to your forked

Re: [PR] [SPARK-47263][SQL] Assign names to the legacy conditions _LEGACY_ERROR_TEMP_13[44-46] [spark]

2024-09-01 Thread via GitHub
MaxGekk commented on PR #46320: URL: https://github.com/apache/spark/pull/46320#issuecomment-2323394999 @PaysonXu Could you remove the text before `What changes were proposed in this pull request?` in PR's description. -- This is an automated message from the Apache Git Service. To respon

Re: [PR] [SPARK-49476][SQL] Fix nullability of base64 function [spark]

2024-09-01 Thread via GitHub
Kimahriman commented on code in PR #47941: URL: https://github.com/apache/spark/pull/47941#discussion_r1740199040 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala: ## @@ -467,6 +467,12 @@ class StringExpressionsSuite extends S

[PR] [SPARK-49477][PYTHON] Improve pandas udf invalid return type error message [spark]

2024-08-30 Thread via GitHub
allisonwang-db opened a new pull request, #47942: URL: https://github.com/apache/spark/pull/47942 ### What changes were proposed in this pull request? This PR improves the error message when the specified return type of a pandas udf mismatch the actual return type. ###

Re: [PR] [SPARK-49465] Fix example props override in helm chart operatorConfiguration.dynamicConfig.data [spark-kubernetes-operator]

2024-08-31 Thread via GitHub
dongjoon-hyun closed pull request #114: [SPARK-49465] Fix example props override in helm chart operatorConfiguration.dynamicConfig.data URL: https://github.com/apache/spark-kubernetes-operator/pull/114 -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] [SPARK-48682][SQL][FOLLOW-UP] Changed initCap behaviour with UTF8_BINARY collation [spark]

2024-08-31 Thread via GitHub
viktorluc-db commented on code in PR #47771: URL: https://github.com/apache/spark/pull/47771#discussion_r1739767821 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java: ## @@ -550,6 +549,152 @@ public static UTF8String toTitleCase(fina

Re: [PR] [WIP][DO NOT REVIEW] dummy PR [spark]

2024-08-31 Thread via GitHub
github-actions[bot] closed pull request #46702: [WIP][DO NOT REVIEW] dummy PR URL: https://github.com/apache/spark/pull/46702 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

  1   2   3   4   5   6   7   8   9   10   >