[GitHub] [spark] EnricoMi commented on a diff in pull request #38312: [SPARK-40819][SQL] Timestamp nanos behaviour regression

EnricoMi commented on code in PR #38312: URL: https://github.com/apache/spark/pull/38312#discussion_r1025407302 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala: ## @@ -1040,6 +1040,14 @@ class ParquetSchemaSuite extends

[GitHub] [spark] awdavidson commented on a diff in pull request #38312: [SPARK-40819][SQL] Timestamp nanos behaviour regression

awdavidson commented on code in PR #38312: URL: https://github.com/apache/spark/pull/38312#discussion_r1025413256 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala: ## @@ -1040,6 +1040,14 @@ class ParquetSchemaSuite extends

[GitHub] [spark] awdavidson commented on a diff in pull request #38312: [SPARK-40819][SQL] Timestamp nanos behaviour regression

awdavidson commented on code in PR #38312: URL: https://github.com/apache/spark/pull/38312#discussion_r1025412863 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala: ## @@ -1040,6 +1040,14 @@ class ParquetSchemaSuite extends

[GitHub] [spark] EnricoMi commented on a diff in pull request #38312: [SPARK-40819][SQL] Timestamp nanos behaviour regression

EnricoMi commented on code in PR #38312: URL: https://github.com/apache/spark/pull/38312#discussion_r1025413969 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala: ## @@ -1040,6 +1040,14 @@ class ParquetSchemaSuite extends

[GitHub] [spark] EnricoMi commented on a diff in pull request #38312: [SPARK-40819][SQL] Timestamp nanos behaviour regression

EnricoMi commented on code in PR #38312: URL: https://github.com/apache/spark/pull/38312#discussion_r1025422089 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala: ## @@ -1040,6 +1040,14 @@ class ParquetSchemaSuite extends

[GitHub] [spark] cloud-fan commented on a diff in pull request #38497: [SPARK-40999] Hint propagation to subqueries

cloud-fan commented on code in PR #38497: URL: https://github.com/apache/spark/pull/38497#discussion_r1025470097 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/EliminateResolvedHint.scala: ## @@ -31,20 +34,35 @@ object EliminateResolvedHint extends

[GitHub] [spark] srowen commented on pull request #38610: [SPARK-41106][SQL] Reduce collection conversion when create AttributeMap

srowen commented on PR #38610: URL: https://github.com/apache/spark/pull/38610#issuecomment-1318859003 Cool, so this has passed on 2.12 and 2.13, seems OK to merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] LuciferYang commented on a diff in pull request #38690: [SPARK-41177][PROTOBUF][TESTS] Fix maven test failed of `protobuf` module

LuciferYang commented on code in PR #38690: URL: https://github.com/apache/spark/pull/38690#discussion_r1025435300 ## connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala: ## @@ -34,9 +34,10 @@ import

[GitHub] [spark] MaxGekk commented on a diff in pull request #37725: [DO-NOT-MERGE] Exceptions without error classes in SQL golden files

MaxGekk commented on code in PR #37725: URL: https://github.com/apache/spark/pull/37725#discussion_r1025206912 ## sql/core/src/test/resources/sql-tests/results/ansi/string-functions.sql.out: ## @@ -5,7 +5,12 @@ select concat_ws() struct<> -- !query output

[GitHub] [spark] cloud-fan commented on a diff in pull request #38497: [SPARK-40999] Hint propagation to subqueries

cloud-fan commented on code in PR #38497: URL: https://github.com/apache/spark/pull/38497#discussion_r1025472753 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala: ## @@ -201,15 +204,17 @@ object RewritePredicateSubquery extends

[GitHub] [spark] dengziming commented on pull request #38659: [SPARK-41114][CONNECT] Support local data for LocalRelation

dengziming commented on PR #38659: URL: https://github.com/apache/spark/pull/38659#issuecomment-1318904845 I resolved the comments and move schema to the arrow batch, there are still some TODOs left behind which I will fix after we all agree this plan. @amaliujia @grundprinzip

[GitHub] [spark] LuciferYang commented on pull request #38610: [SPARK-41106][SQL] Reduce collection conversion when create AttributeMap

LuciferYang commented on PR #38610: URL: https://github.com/apache/spark/pull/38610#issuecomment-1318927149 Thanks @srowen ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] cloud-fan commented on pull request #38692: [SPARK-41183][SQL] Add an extension API to do plan normalization for caching

cloud-fan commented on PR #38692: URL: https://github.com/apache/spark/pull/38692#issuecomment-1318943621 @viirya @wangyum @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] LuciferYang commented on pull request #38610: [SPARK-41106][SQL] Reduce collection conversion when create AttributeMap

LuciferYang commented on PR #38610: URL: https://github.com/apache/spark/pull/38610#issuecomment-1318907594 Yes, both Scala 2.12 and Scala 2.13 have passed the test :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] srowen closed pull request #38610: [SPARK-41106][SQL] Reduce collection conversion when create AttributeMap

srowen closed pull request #38610: [SPARK-41106][SQL] Reduce collection conversion when create AttributeMap URL: https://github.com/apache/spark/pull/38610 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] srowen commented on pull request #38610: [SPARK-41106][SQL] Reduce collection conversion when create AttributeMap

srowen commented on PR #38610: URL: https://github.com/apache/spark/pull/38610#issuecomment-1318924610 Merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] cloud-fan opened a new pull request, #38692: [SPARK-41183][SQL] Add an extension API to do plan normalization for caching

cloud-fan opened a new pull request, #38692: URL: https://github.com/apache/spark/pull/38692 ### What changes were proposed in this pull request? Today, Spark is very conservative and uses the analyzed plan instead of the optimized plan as the cache key. Many cache

[GitHub] [spark] MaxGekk commented on pull request #37725: [DO-NOT-MERGE] Exceptions without error classes in SQL golden files

MaxGekk commented on PR #37725: URL: https://github.com/apache/spark/pull/37725#issuecomment-1318946953 @panbingkun @LuciferYang @itholic @cloud-fan @srielau @anchovYu @entong I changed the mechanism of generating SQL golden files in this PR to detect the exceptions that haven't been

[GitHub] [spark] grundprinzip opened a new pull request, #38693: Homogenize the python proto version

grundprinzip opened a new pull request, #38693: URL: https://github.com/apache/spark/pull/38693 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ###

[GitHub] [spark] amaliujia commented on a diff in pull request #38678: [SPARK-41164][CONNECT] Update relations.proto to follow Connect proto development guide

amaliujia commented on code in PR #38678: URL: https://github.com/apache/spark/pull/38678#discussion_r1025551926 ## connector/connect/src/main/protobuf/spark/connect/relations.proto: ## @@ -106,24 +113,39 @@ message Project { // // For example, `SELECT ABS(-1)` is valid

[GitHub] [spark] amaliujia commented on a diff in pull request #38678: [SPARK-41164][CONNECT] Update relations.proto to follow Connect proto development guide

amaliujia commented on code in PR #38678: URL: https://github.com/apache/spark/pull/38678#discussion_r1025552911 ## connector/connect/src/main/protobuf/spark/connect/relations.proto: ## @@ -106,24 +113,39 @@ message Project { // // For example, `SELECT ABS(-1)` is valid

[GitHub] [spark] amaliujia commented on pull request #38694: [SPARK-41184][CONNECT] Disable flakey Fill.NA tests

amaliujia commented on PR #38694: URL: https://github.com/apache/spark/pull/38694#issuecomment-1319136680 LGTM cc @zhengruifeng -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] vitas commented on pull request #38262: [SPARK-40801][BUILD] Upgrade `Apache commons-text` to 1.10

vitas commented on PR #38262: URL: https://github.com/apache/spark/pull/38262#issuecomment-1319168813 when 3.3.2 comes out? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] MaxGekk closed pull request #38647: [SPARK-41133][SQL] Integrate `UNSCALED_VALUE_TOO_LARGE_FOR_PRECISION` into `NUMERIC_VALUE_OUT_OF_RANGE`

MaxGekk closed pull request #38647: [SPARK-41133][SQL] Integrate `UNSCALED_VALUE_TOO_LARGE_FOR_PRECISION` into `NUMERIC_VALUE_OUT_OF_RANGE` URL: https://github.com/apache/spark/pull/38647 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] MaxGekk commented on pull request #38647: [SPARK-41133][SQL] Integrate `UNSCALED_VALUE_TOO_LARGE_FOR_PRECISION` into `NUMERIC_VALUE_OUT_OF_RANGE`

MaxGekk commented on PR #38647: URL: https://github.com/apache/spark/pull/38647#issuecomment-1319038961 +1, LGTM. Merging to master. Thank you, @itholic and @srielau for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] gengliangwang commented on a diff in pull request #38567: [SPARK-41054][UI][CORE] Support RocksDB as KVStore in live UI

gengliangwang commented on code in PR #38567: URL: https://github.com/apache/spark/pull/38567#discussion_r1025583467 ## core/src/main/scala/org/apache/spark/status/KVUtils.scala: ## @@ -80,6 +89,44 @@ private[spark] object KVUtils extends Logging { db } + def

[GitHub] [spark] hvanhovell opened a new pull request, #38694: [SPARK-41184][CONNECT] Disable flakey Fill.NA tests

hvanhovell opened a new pull request, #38694: URL: https://github.com/apache/spark/pull/38694 ### What changes were proposed in this pull request? Disable Connect's Pyhon Fill.NA tests because they are flakey. ### Why are the changes needed? Connect's Pyhon Fill.NA tests because

[GitHub] [spark] amaliujia commented on pull request #38693: Homogenize the python proto version

amaliujia commented on PR #38693: URL: https://github.com/apache/spark/pull/38693#issuecomment-1319155192 Also need to make the Scala side version consistent https://github.com/apache/spark/blob/master/connector/connect/pom.xml#L35? -- This is an automated message from the Apache Git

[GitHub] [spark] amaliujia commented on pull request #38681: [SPARK-41165][CONNECT] Avoid hangs in the arrow collect code path

amaliujia commented on PR #38681: URL: https://github.com/apache/spark/pull/38681#issuecomment-1319030069 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] amaliujia commented on a diff in pull request #38678: [SPARK-41164][CONNECT] Update relations.proto to follow Connect proto development guide

amaliujia commented on code in PR #38678: URL: https://github.com/apache/spark/pull/38678#discussion_r1025551926 ## connector/connect/src/main/protobuf/spark/connect/relations.proto: ## @@ -106,24 +113,39 @@ message Project { // // For example, `SELECT ABS(-1)` is valid

[GitHub] [spark] mridulm commented on pull request #38567: [SPARK-41054][UI][CORE] Support RocksDB as KVStore in live UI

mridulm commented on PR #38567: URL: https://github.com/apache/spark/pull/38567#issuecomment-1319148903 Sorry for the delay, I will try to review this later this week ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] amaliujia commented on pull request #38693: Homogenize the python proto version

amaliujia commented on PR #38693: URL: https://github.com/apache/spark/pull/38693#issuecomment-1319148756 @grundprinzip You need to re-generated the protobuf for Python side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] amaliujia commented on pull request #38638: [SPARK-41122][CONNECT] Explain API can support different modes

amaliujia commented on PR #38638: URL: https://github.com/apache/spark/pull/38638#issuecomment-1319362671 @HyukjinKwon @cloud-fan @grundprinzip @zhengruifeng please take another look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] amaliujia commented on a diff in pull request #38638: [SPARK-41122][CONNECT] Explain API can support different modes

amaliujia commented on code in PR #38638: URL: https://github.com/apache/spark/pull/38638#discussion_r1025851202 ## connector/connect/src/main/protobuf/spark/connect/base.proto: ## @@ -38,16 +38,50 @@ message Plan { } } +// Explains the input plan based on a configurable

[GitHub] [spark] bjornjorgensen commented on pull request #38262: [SPARK-40801][BUILD] Upgrade `Apache commons-text` to 1.10

bjornjorgensen commented on PR #38262: URL: https://github.com/apache/spark/pull/38262#issuecomment-1319181705 @vitas I think it is best to ask questions like that on the mailing list https://spark.apache.org/community.html under Mailing lists -- This is an automated message from the

<    1   2