[GitHub] [spark] LuciferYang commented on a diff in pull request #38369: [SPARK-40895][BUILD] Upgrade arrow to 10.0.0

2022-10-23 Thread GitBox
LuciferYang commented on code in PR #38369: URL: https://github.com/apache/spark/pull/38369#discussion_r1002902510 ## project/SparkBuild.scala: ## @@ -291,6 +291,7 @@ object SparkBuild extends PomBuild { // Google Mirror of Maven Central, placed first so that it's used

[GitHub] [spark] awdavidson commented on a diff in pull request #38312: [SPARK-40819][SQL] Timestamp nanos behaviour regression

2022-10-23 Thread GitBox
awdavidson commented on code in PR #38312: URL: https://github.com/apache/spark/pull/38312#discussion_r1002902562 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala: ## @@ -271,6 +271,10 @@ class

[GitHub] [spark] LuciferYang commented on a diff in pull request #38369: [SPARK-40895][BUILD] Upgrade arrow to 10.0.0

2022-10-23 Thread GitBox
LuciferYang commented on code in PR #38369: URL: https://github.com/apache/spark/pull/38369#discussion_r1002902416 ## pom.xml: ## @@ -319,6 +319,17 @@ + Review Comment: for test, will revert later -- This is an automated message from the Apache Git

[GitHub] [spark] MaxGekk commented on a diff in pull request #38364: [SPARK-40891][SQL][TESTS] Check error classes in TableIdentifierParserSuite

2022-10-23 Thread GitBox
MaxGekk commented on code in PR #38364: URL: https://github.com/apache/spark/pull/38364#discussion_r1002902138 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala: ## @@ -290,9 +290,35 @@ class TableIdentifierParserSuite extends

[GitHub] [spark] cloud-fan commented on a diff in pull request #38320: [SPARK-40857] [CONNECT] Enable configurable GPRC Interceptors.

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38320: URL: https://github.com/apache/spark/pull/38320#discussion_r1002901217 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala: ## @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache

[GitHub] [spark] cloud-fan commented on a diff in pull request #38320: [SPARK-40857] [CONNECT] Enable configurable GPRC Interceptors.

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38320: URL: https://github.com/apache/spark/pull/38320#discussion_r1002900934 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala: ## @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache

[GitHub] [spark] cloud-fan commented on a diff in pull request #38320: [SPARK-40857] [CONNECT] Enable configurable GPRC Interceptors.

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38320: URL: https://github.com/apache/spark/pull/38320#discussion_r1002900997 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala: ## @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache

[GitHub] [spark] cloud-fan commented on a diff in pull request #38320: [SPARK-40857] [CONNECT] Enable configurable GPRC Interceptors.

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38320: URL: https://github.com/apache/spark/pull/38320#discussion_r1002900490 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala: ## @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache

[GitHub] [spark] LuciferYang opened a new pull request, #38369: [SPARK-40895][BUILD] Upgrade arrow to 10.0.0

2022-10-23 Thread GitBox
LuciferYang opened a new pull request, #38369: URL: https://github.com/apache/spark/pull/38369 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ###

[GitHub] [spark] panbingkun commented on a diff in pull request #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes

2022-10-23 Thread GitBox
panbingkun commented on code in PR #38332: URL: https://github.com/apache/spark/pull/38332#discussion_r1002898122 ## core/src/main/resources/error/error-classes.json: ## @@ -143,6 +143,11 @@ "Offset expression must be a literal." ] }, +

[GitHub] [spark] rangadi commented on a diff in pull request #38344: [SPARK-40777][SQL][PROTOBUF] Protobuf import support and move error-classes.

2022-10-23 Thread GitBox
rangadi commented on code in PR #38344: URL: https://github.com/apache/spark/pull/38344#discussion_r1002893438 ## connector/protobuf/pom.xml: ## @@ -123,6 +123,7 @@ com.google.protobuf:protoc:${protobuf.version} ${protobuf.version} +

[GitHub] [spark] MaxGekk commented on a diff in pull request #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes

2022-10-23 Thread GitBox
MaxGekk commented on code in PR #38332: URL: https://github.com/apache/spark/pull/38332#discussion_r1002891939 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala: ## @@ -268,15 +269,17 @@ abstract class HashExpression[E] extends Expression {

[GitHub] [spark] cloud-fan commented on a diff in pull request #38345: [SPARK-40879][CONNECT] Support Join UsingColumns in proto

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38345: URL: https://github.com/apache/spark/pull/38345#discussion_r100258 ## connector/connect/src/main/protobuf/spark/connect/relations.proto: ## @@ -109,6 +109,7 @@ message Join { Relation right = 2; Expression join_condition = 3;

[GitHub] [spark] itholic commented on a diff in pull request #38177: [SPARK-40663][SQL](Final) Migrate execution errors onto error classes

2022-10-23 Thread GitBox
itholic commented on code in PR #38177: URL: https://github.com/apache/spark/pull/38177#discussion_r1002888243 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala: ## @@ -1124,8 +1124,9 @@ class StringExpressionsSuite extends

[GitHub] [spark] amaliujia commented on a diff in pull request #38345: [SPARK-40879][CONNECT] Support Join UsingColumns in proto

2022-10-23 Thread GitBox
amaliujia commented on code in PR #38345: URL: https://github.com/apache/spark/pull/38345#discussion_r1002886307 ## connector/connect/src/main/protobuf/spark/connect/relations.proto: ## @@ -109,6 +109,7 @@ message Join { Relation right = 2; Expression join_condition = 3;

[GitHub] [spark] amaliujia commented on a diff in pull request #38301: [SPARK-40836][CONNECT] AnalyzeResult should use struct for schema

2022-10-23 Thread GitBox
amaliujia commented on code in PR #38301: URL: https://github.com/apache/spark/pull/38301#discussion_r1002886560 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/DataTypeProtoConverter.scala: ## @@ -50,11 +50,27 @@ object DataTypeProtoConverter {

[GitHub] [spark] amaliujia commented on a diff in pull request #38345: [SPARK-40879][CONNECT] Support Join UsingColumns in proto

2022-10-23 Thread GitBox
amaliujia commented on code in PR #38345: URL: https://github.com/apache/spark/pull/38345#discussion_r1002886307 ## connector/connect/src/main/protobuf/spark/connect/relations.proto: ## @@ -109,6 +109,7 @@ message Join { Relation right = 2; Expression join_condition = 3;

[GitHub] [spark] cloud-fan commented on a diff in pull request #38301: [SPARK-40836][CONNECT] AnalyzeResult should use struct for schema

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38301: URL: https://github.com/apache/spark/pull/38301#discussion_r1002886171 ## python/pyspark/sql/connect/client.py: ## @@ -151,7 +151,44 @@ def _to_pandas(self, plan: pb2.Plan) -> Optional[pandas.DataFrame]:

[GitHub] [spark] cloud-fan commented on a diff in pull request #38301: [SPARK-40836][CONNECT] AnalyzeResult should use struct for schema

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38301: URL: https://github.com/apache/spark/pull/38301#discussion_r1002886025 ## python/pyspark/sql/connect/client.py: ## @@ -151,7 +151,44 @@ def _to_pandas(self, plan: pb2.Plan) -> Optional[pandas.DataFrame]:

[GitHub] [spark] amaliujia commented on a diff in pull request #38301: [SPARK-40836][CONNECT] AnalyzeResult should use struct for schema

2022-10-23 Thread GitBox
amaliujia commented on code in PR #38301: URL: https://github.com/apache/spark/pull/38301#discussion_r1002885859 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/DataTypeProtoConverter.scala: ## @@ -50,11 +50,27 @@ object DataTypeProtoConverter {

[GitHub] [spark] cloud-fan commented on a diff in pull request #38301: [SPARK-40836][CONNECT] AnalyzeResult should use struct for schema

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38301: URL: https://github.com/apache/spark/pull/38301#discussion_r1002885552 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/DataTypeProtoConverter.scala: ## @@ -50,11 +50,27 @@ object DataTypeProtoConverter {

[GitHub] [spark] cloud-fan commented on a diff in pull request #38312: [SPARK-40819][SQL] Timestamp nanos behaviour regression

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38312: URL: https://github.com/apache/spark/pull/38312#discussion_r1002884323 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala: ## @@ -271,6 +271,10 @@ class

[GitHub] [spark] cloud-fan commented on a diff in pull request #38345: [SPARK-40879][CONNECT] Support Join UsingColumns in proto

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38345: URL: https://github.com/apache/spark/pull/38345#discussion_r1002881870 ## connector/connect/src/main/protobuf/spark/connect/relations.proto: ## @@ -109,6 +109,7 @@ message Join { Relation right = 2; Expression join_condition = 3;

[GitHub] [spark] MaxGekk commented on a diff in pull request #38344: [SPARK-40777][SQL][PROTOBUF] Protobuf import support and move error-classes.

2022-10-23 Thread GitBox
MaxGekk commented on code in PR #38344: URL: https://github.com/apache/spark/pull/38344#discussion_r1002880566 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -3210,4 +3211,183 @@ private[sql] object QueryCompilationErrors extends

[GitHub] [spark] MaxGekk commented on pull request #38170: [WIP][SPARK-40663][SQL] Migrate execution errors onto error classes: _LEGACY_ERROR_TEMP_2201-2225

2022-10-23 Thread GitBox
MaxGekk commented on PR #38170: URL: https://github.com/apache/spark/pull/38170#issuecomment-1288415776 @itholic Could you fix the test failures: ``` [info] - SPARK-34057: UnresolvedTableOrView should retain sql text position *** FAILED *** (139 milliseconds) [info] Expected

[GitHub] [spark] MaxGekk commented on pull request #38175: [WIP][SPARK-40663][SQL] Migrate execution errors onto error classes: _LEGACY_ERROR_TEMP_2251-2275

2022-10-23 Thread GitBox
MaxGekk commented on PR #38175: URL: https://github.com/apache/spark/pull/38175#issuecomment-1288414521 @itholic Could you resolve conflicts, please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] MaxGekk commented on pull request #38177: [SPARK-40663][SQL](Final) Migrate execution errors onto error classes

2022-10-23 Thread GitBox
MaxGekk commented on PR #38177: URL: https://github.com/apache/spark/pull/38177#issuecomment-1288414141 ping @itholic -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] MaxGekk commented on pull request #38360: [SPARK-40811][SQL][TESTS] Use `checkError()` to intercept `ParseException`: phase 2

2022-10-23 Thread GitBox
MaxGekk commented on PR #38360: URL: https://github.com/apache/spark/pull/38360#issuecomment-1288411700 cc @panbingkun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] HeartSaVioR commented on pull request #38368: [SPARK-40821][SQL][SS][FOLLOWUP] Fix available version for new function window_time

2022-10-23 Thread GitBox
HeartSaVioR commented on PR #38368: URL: https://github.com/apache/spark/pull/38368#issuecomment-1288397473 cc. @cloud-fan @alex-balikov -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HeartSaVioR opened a new pull request, #38368: [SPARK-40821][SQL][SS][FOLLOWUP] Fix available version for new function window_time

2022-10-23 Thread GitBox
HeartSaVioR opened a new pull request, #38368: URL: https://github.com/apache/spark/pull/38368 ### What changes were proposed in this pull request? This PR fixes the incorrect available version for new function `window_time` to 3.4.0 which is upcoming release for master branch.

[GitHub] [spark] sandeep-katta commented on pull request #26777: [SPARK-30134][SQL] Support DELETE JAR feature in SPARK

2022-10-23 Thread GitBox
sandeep-katta commented on PR #26777: URL: https://github.com/apache/spark/pull/26777#issuecomment-1288382078 @diaolimin consensus were reached not to add new API like delete jar, so it was not merged. And also it is not production ready as this PR targets only one part of the umbrella

[GitHub] [spark] diaolimin commented on pull request #26777: [SPARK-30134][SQL] Support DELETE JAR feature in SPARK

2022-10-23 Thread GitBox
diaolimin commented on PR #26777: URL: https://github.com/apache/spark/pull/26777#issuecomment-1288374130 @sandeep-katta I want to ask what is the problem? Can it be used in a production environment? -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] sandeep-katta commented on pull request #26777: [SPARK-30134][SQL] Support DELETE JAR feature in SPARK

2022-10-23 Thread GitBox
sandeep-katta commented on PR #26777: URL: https://github.com/apache/spark/pull/26777#issuecomment-1288371386 @diaolimin unfortunately this fix is not merged, so you need to restart the thrift server to update the jar definition -- This is an automated message from the Apache Git

[GitHub] [spark] panbingkun closed pull request #38367: [WIP][MINOR] Remove unused import in StatFunctions

2022-10-23 Thread GitBox
panbingkun closed pull request #38367: [WIP][MINOR] Remove unused import in StatFunctions URL: https://github.com/apache/spark/pull/38367 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls

2022-10-23 Thread GitBox
HeartSaVioR commented on code in PR #38361: URL: https://github.com/apache/spark/pull/38361#discussion_r1002857134 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala: ## @@ -292,53 +292,59 @@ object ResolveWindowTime extends

[GitHub] [spark] panbingkun commented on pull request #38367: [WIP][MINOR] Remove unused import in StatFunctions

2022-10-23 Thread GitBox
panbingkun commented on PR #38367: URL: https://github.com/apache/spark/pull/38367#issuecomment-1288365900 > this was resolved in [f7eee09](https://github.com/apache/spark/commit/f7eee0950493ede83f5f00be2030cb8111ae6aa1) Ok, i close it. -- This is an automated message from the

[GitHub] [spark] cloud-fan commented on a diff in pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38361: URL: https://github.com/apache/spark/pull/38361#discussion_r1002855960 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala: ## @@ -292,53 +292,59 @@ object ResolveWindowTime extends

[GitHub] [spark] diaolimin commented on pull request #26777: [SPARK-30134][SQL] Support DELETE JAR feature in SPARK

2022-10-23 Thread GitBox
diaolimin commented on PR #26777: URL: https://github.com/apache/spark/pull/26777#issuecomment-1288362205 @sandeep-katta I encountered this problem. Which version is planned to be merged? -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] LuciferYang commented on a diff in pull request #38366: [SPARK-40893][INFRA] Upgrade to use setup-java v3

2022-10-23 Thread GitBox
LuciferYang commented on code in PR #38366: URL: https://github.com/apache/spark/pull/38366#discussion_r1002854458 ## .github/workflows/benchmark.yml: ## @@ -105,9 +105,11 @@ jobs: run: cd tpcds-kit/tools && make OS=LINUX - name: Install Java ${{

[GitHub] [spark] LuciferYang closed pull request #38366: [SPARK-40893][INFRA] Upgrade to use setup-java v3

2022-10-23 Thread GitBox
LuciferYang closed pull request #38366: [SPARK-40893][INFRA] Upgrade to use setup-java v3 URL: https://github.com/apache/spark/pull/38366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] Yikun commented on a diff in pull request #38366: [SPARK-40893][INFRA] Upgrade to use setup-java v3

2022-10-23 Thread GitBox
Yikun commented on code in PR #38366: URL: https://github.com/apache/spark/pull/38366#discussion_r1002853595 ## .github/workflows/benchmark.yml: ## @@ -105,9 +105,11 @@ jobs: run: cd tpcds-kit/tools && make OS=LINUX - name: Install Java ${{

[GitHub] [spark] zhengruifeng commented on pull request #38367: [WIP][MINOR] Remove unused import in StatFunctions

2022-10-23 Thread GitBox
zhengruifeng commented on PR #38367: URL: https://github.com/apache/spark/pull/38367#issuecomment-1288356731 this was resolved in https://github.com/apache/spark/commit/f7eee0950493ede83f5f00be2030cb8111ae6aa1 -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] diaolimin commented on pull request #26777: [SPARK-30134][SQL] Support DELETE JAR feature in SPARK

2022-10-23 Thread GitBox
diaolimin commented on PR #26777: URL: https://github.com/apache/spark/pull/26777#issuecomment-1288356113 > let me make it clear > > This PR aims at solving following use case,. > > user can update the jar definition and load to spark without restarting the thrift server. >

[GitHub] [spark] cloud-fan closed pull request #38260: [SPARK-40800][SQL] Always inline expressions in OptimizeOneRowRelationSubquery

2022-10-23 Thread GitBox
cloud-fan closed pull request #38260: [SPARK-40800][SQL] Always inline expressions in OptimizeOneRowRelationSubquery URL: https://github.com/apache/spark/pull/38260 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] cloud-fan commented on pull request #38260: [SPARK-40800][SQL] Always inline expressions in OptimizeOneRowRelationSubquery

2022-10-23 Thread GitBox
cloud-fan commented on PR #38260: URL: https://github.com/apache/spark/pull/38260#issuecomment-1288355575 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls

2022-10-23 Thread GitBox
HeartSaVioR commented on code in PR #38361: URL: https://github.com/apache/spark/pull/38361#discussion_r1002851334 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala: ## @@ -292,53 +292,59 @@ object ResolveWindowTime extends

[GitHub] [spark] panbingkun opened a new pull request, #38367: [WIP][MINOR] Remove unused import in StatFunctions

2022-10-23 Thread GitBox
panbingkun opened a new pull request, #38367: URL: https://github.com/apache/spark/pull/38367 ### What changes were proposed in this pull request? Remove unused import in `StatFunctions.scala`. ### Why are the changes needed? Cleanup ### Does this PR introduce _any_

[GitHub] [spark] amaliujia commented on a diff in pull request #38347: [SPARK-40883][CONNECT] Support Range in Connect proto

2022-10-23 Thread GitBox
amaliujia commented on code in PR #38347: URL: https://github.com/apache/spark/pull/38347#discussion_r1002851496 ## connector/connect/src/main/protobuf/spark/connect/relations.proto: ## @@ -207,3 +208,23 @@ message Sample { int64 seed = 1; } } + +// Relation of type

[GitHub] [spark] cloud-fan closed pull request #38135: [SPARK-36114][SQL] Support subqueries with correlated non-equality predicates

2022-10-23 Thread GitBox
cloud-fan closed pull request #38135: [SPARK-36114][SQL] Support subqueries with correlated non-equality predicates URL: https://github.com/apache/spark/pull/38135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] cloud-fan commented on pull request #38135: [SPARK-36114][SQL] Support subqueries with correlated non-equality predicates

2022-10-23 Thread GitBox
cloud-fan commented on PR #38135: URL: https://github.com/apache/spark/pull/38135#issuecomment-1288354293 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls

2022-10-23 Thread GitBox
HeartSaVioR commented on code in PR #38361: URL: https://github.com/apache/spark/pull/38361#discussion_r1002851334 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala: ## @@ -292,53 +292,59 @@ object ResolveWindowTime extends

[GitHub] [spark] amaliujia commented on a diff in pull request #38347: [SPARK-40883][CONNECT] Support Range in Connect proto

2022-10-23 Thread GitBox
amaliujia commented on code in PR #38347: URL: https://github.com/apache/spark/pull/38347#discussion_r1002850714 ## connector/connect/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectSessionBasedSuite.scala: ## @@ -32,7 +35,9 @@ trait

[GitHub] [spark] amaliujia commented on a diff in pull request #38347: [SPARK-40883][CONNECT] Support Range in Connect proto

2022-10-23 Thread GitBox
amaliujia commented on code in PR #38347: URL: https://github.com/apache/spark/pull/38347#discussion_r1002850714 ## connector/connect/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectSessionBasedSuite.scala: ## @@ -32,7 +35,9 @@ trait

[GitHub] [spark] cloud-fan commented on a diff in pull request #38347: [SPARK-40883][CONNECT] Support Range in Connect proto

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38347: URL: https://github.com/apache/spark/pull/38347#discussion_r1002850206 ## connector/connect/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectSessionBasedSuite.scala: ## @@ -32,7 +35,9 @@ trait

[GitHub] [spark] cloud-fan commented on a diff in pull request #38347: [SPARK-40883][CONNECT] Support Range in Connect proto

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38347: URL: https://github.com/apache/spark/pull/38347#discussion_r1002849988 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -94,6 +96,25 @@ class SparkConnectPlanner(plan:

[GitHub] [spark] cloud-fan commented on a diff in pull request #38347: [SPARK-40883][CONNECT] Support Range in Connect proto

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38347: URL: https://github.com/apache/spark/pull/38347#discussion_r1002849365 ## connector/connect/src/main/protobuf/spark/connect/relations.proto: ## @@ -207,3 +208,23 @@ message Sample { int64 seed = 1; } } + +// Relation of type

[GitHub] [spark] amaliujia commented on a diff in pull request #38276: [SPARK-40812][CONNECT] Add Deduplicate to Connect proto and DSL

2022-10-23 Thread GitBox
amaliujia commented on code in PR #38276: URL: https://github.com/apache/spark/pull/38276#discussion_r1002848070 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -91,6 +93,37 @@ class SparkConnectPlanner(plan:

[GitHub] [spark] LuciferYang commented on pull request #38366: [SPARK-40893][INFRA] Upgrade to use setup-java v3

2022-10-23 Thread GitBox
LuciferYang commented on PR #38366: URL: https://github.com/apache/spark/pull/38366#issuecomment-1288348240 Will update pr description later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] amaliujia commented on a diff in pull request #38276: [SPARK-40812][CONNECT] Add Deduplicate to Connect proto and DSL

2022-10-23 Thread GitBox
amaliujia commented on code in PR #38276: URL: https://github.com/apache/spark/pull/38276#discussion_r1002848070 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -91,6 +93,37 @@ class SparkConnectPlanner(plan:

[GitHub] [spark] amaliujia commented on a diff in pull request #38276: [SPARK-40812][CONNECT] Add Deduplicate to Connect proto and DSL

2022-10-23 Thread GitBox
amaliujia commented on code in PR #38276: URL: https://github.com/apache/spark/pull/38276#discussion_r1002848070 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -91,6 +93,37 @@ class SparkConnectPlanner(plan:

[GitHub] [spark] LuciferYang opened a new pull request, #38366: [SPARK-40893][INFRA] Upgrade to use setup-java v3

2022-10-23 Thread GitBox
LuciferYang opened a new pull request, #38366: URL: https://github.com/apache/spark/pull/38366 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ###

[GitHub] [spark] amaliujia commented on a diff in pull request #38276: [SPARK-40812][CONNECT] Add Deduplicate to Connect proto and DSL

2022-10-23 Thread GitBox
amaliujia commented on code in PR #38276: URL: https://github.com/apache/spark/pull/38276#discussion_r1002848070 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -91,6 +93,37 @@ class SparkConnectPlanner(plan:

[GitHub] [spark] cloud-fan commented on a diff in pull request #38361: [SPARK-40892][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38361: URL: https://github.com/apache/spark/pull/38361#discussion_r1002847203 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala: ## @@ -292,53 +292,59 @@ object ResolveWindowTime extends

[GitHub] [spark] LuciferYang commented on pull request #38363: [SPARK-40851][INFRA ][SQL][TESTS][3.3] Make GA run successfully with the latest Java 8/11/17

2022-10-23 Thread GitBox
LuciferYang commented on PR #38363: URL: https://github.com/apache/spark/pull/38363#issuecomment-1288342704 waiting ci -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] LuciferYang commented on pull request #38365: [SPARK-40851][INFRA ][SQL][TESTS][3.2] Make GA run successfully with the latest Java 8/11/17

2022-10-23 Thread GitBox
LuciferYang commented on PR #38365: URL: https://github.com/apache/spark/pull/38365#issuecomment-1288342632 waiting ci -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] LuciferYang commented on a diff in pull request #38317: [SPARK-40851][INFRA ][SQL][TESTS] Make GA run successfully with the latest Java 8/11/17

2022-10-23 Thread GitBox
LuciferYang commented on code in PR #38317: URL: https://github.com/apache/spark/pull/38317#discussion_r1002846194 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala: ## @@ -937,16 +937,16 @@ class DateTimeUtilsSuite extends SparkFunSuite

[GitHub] [spark] cloud-fan closed pull request #38327: [SPARK-40812][CONNECT][PYTHON][FOLLOW-UP] Improve Deduplicate in Python client

2022-10-23 Thread GitBox
cloud-fan closed pull request #38327: [SPARK-40812][CONNECT][PYTHON][FOLLOW-UP] Improve Deduplicate in Python client URL: https://github.com/apache/spark/pull/38327 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] cloud-fan commented on pull request #38327: [SPARK-40812][CONNECT][PYTHON][FOLLOW-UP] Improve Deduplicate in Python client

2022-10-23 Thread GitBox
cloud-fan commented on PR #38327: URL: https://github.com/apache/spark/pull/38327#issuecomment-1288340906 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] LuciferYang opened a new pull request, #38365: [SPARK-40851][INFRA ][SQL][TESTS][3.2] Make GA run successfully with the latest Java 8/11/17

2022-10-23 Thread GitBox
LuciferYang opened a new pull request, #38365: URL: https://github.com/apache/spark/pull/38365 ### What changes were proposed in this pull request? The main change of this pr as follows: - Replace `Antarctica/Vostok` to `Asia/Urumqi` in Spark code - Replace `Europe/Amsterdam` to

[GitHub] [spark] panbingkun opened a new pull request, #38364: [SPARK-40891][SQL][TESTS] Check error classes in TableIdentifierParserSuite

2022-10-23 Thread GitBox
panbingkun opened a new pull request, #38364: URL: https://github.com/apache/spark/pull/38364 ### What changes were proposed in this pull request? his PR aims to replace 'intercept' with 'Check error classes' in TableIdentifierParserSuite. ### Why are the changes needed? The

[GitHub] [spark] LuciferYang opened a new pull request, #38363: [SPARK-40851][INFRA ][SQL][TESTS][3.3] Make GA run successfully with the latest Java 8/11/17

2022-10-23 Thread GitBox
LuciferYang opened a new pull request, #38363: URL: https://github.com/apache/spark/pull/38363 ### What changes were proposed in this pull request? The main change of this pr as follows: - Replace `Antarctica/Vostok` to `Asia/Urumqi` in Spark code - Replace `Europe/Amsterdam` to

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38327: [SPARK-40812][CONNECT][PYTHON][FOLLOW-UP] Improve Deduplicate in Python client

2022-10-23 Thread GitBox
HyukjinKwon commented on code in PR #38327: URL: https://github.com/apache/spark/pull/38327#discussion_r1002841262 ## python/pyspark/sql/connect/dataframe.py: ## @@ -157,11 +157,44 @@ def coalesce(self, num_partitions: int) -> "DataFrame": def describe(self, cols:

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38276: [SPARK-40812][CONNECT] Add Deduplicate to Connect proto and DSL

2022-10-23 Thread GitBox
HyukjinKwon commented on code in PR #38276: URL: https://github.com/apache/spark/pull/38276#discussion_r1002839336 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -91,6 +93,37 @@ class SparkConnectPlanner(plan:

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38276: [SPARK-40812][CONNECT] Add Deduplicate to Connect proto and DSL

2022-10-23 Thread GitBox
HyukjinKwon commented on code in PR #38276: URL: https://github.com/apache/spark/pull/38276#discussion_r1002838067 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -91,6 +93,37 @@ class SparkConnectPlanner(plan:

[GitHub] [spark] cloud-fan commented on a diff in pull request #38276: [SPARK-40812][CONNECT] Add Deduplicate to Connect proto and DSL

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38276: URL: https://github.com/apache/spark/pull/38276#discussion_r1002833441 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -91,6 +93,37 @@ class SparkConnectPlanner(plan:

[GitHub] [spark] cloud-fan commented on a diff in pull request #38276: [SPARK-40812][CONNECT] Add Deduplicate to Connect proto and DSL

2022-10-23 Thread GitBox
cloud-fan commented on code in PR #38276: URL: https://github.com/apache/spark/pull/38276#discussion_r1002832296 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -91,6 +93,37 @@ class SparkConnectPlanner(plan:

[GitHub] [spark] HyukjinKwon commented on pull request #38344: [SPARK-40777][SQL][PROTOBUF] Protobuf import support and move error-classes.

2022-10-23 Thread GitBox
HyukjinKwon commented on PR #38344: URL: https://github.com/apache/spark/pull/38344#issuecomment-1288323940 cc @MaxGekk FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] HyukjinKwon closed pull request #38362: [SPARK-40880][SQL][FOLLOW-UP] Remove unused imports

2022-10-23 Thread GitBox
HyukjinKwon closed pull request #38362: [SPARK-40880][SQL][FOLLOW-UP] Remove unused imports URL: https://github.com/apache/spark/pull/38362 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] LuciferYang commented on a diff in pull request #38317: [SPARK-40851][INFRA ][SQL][TESTS] Make GA run successfully with the latest Java 8/11/17

2022-10-23 Thread GitBox
LuciferYang commented on code in PR #38317: URL: https://github.com/apache/spark/pull/38317#discussion_r1002829317 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala: ## @@ -937,16 +937,16 @@ class DateTimeUtilsSuite extends SparkFunSuite

[GitHub] [spark] zhengruifeng commented on pull request #38362: [SPARK-40880][SQL][FOLLOW-UP] Remove unused imports

2022-10-23 Thread GitBox
zhengruifeng commented on PR #38362: URL: https://github.com/apache/spark/pull/38362#issuecomment-1288321234 both https://github.com/apache/spark/pull/38346 and https://github.com/apache/spark/pull/38340 had green CI, but merge them together cause the unused imports -- This is an

[GitHub] [spark] HyukjinKwon commented on pull request #38362: [SPARK-40880][SQL][FOLLOW-UP] Remove unused imports

2022-10-23 Thread GitBox
HyukjinKwon commented on PR #38362: URL: https://github.com/apache/spark/pull/38362#issuecomment-1288321205 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] zhengruifeng commented on pull request #38346: [SPARK-40880][SQL] Reimplement `summary` with dataframe operations

2022-10-23 Thread GitBox
zhengruifeng commented on PR #38346: URL: https://github.com/apache/spark/pull/38346#issuecomment-1288320408 CI is green in this PR, but it may have conflict with the previous one https://github.com/apache/spark/pull/38340 then the build in master is broken, I fix it in

[GitHub] [spark] zhengruifeng opened a new pull request, #38362: [SPARK-40880][SQL][FOLLOW-UP] Remove unused imports

2022-10-23 Thread GitBox
zhengruifeng opened a new pull request, #38362: URL: https://github.com/apache/spark/pull/38362 ### What changes were proposed in this pull request? remove unused imports ### Why are the changes needed? ``` [error]

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38276: [SPARK-40812][CONNECT] Add Deduplicate to Connect proto and DSL

2022-10-23 Thread GitBox
HyukjinKwon commented on code in PR #38276: URL: https://github.com/apache/spark/pull/38276#discussion_r1002828551 ## connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -91,6 +93,37 @@ class SparkConnectPlanner(plan:

[GitHub] [spark] HeartSaVioR closed pull request #38313: [SPARK-40849][SS] Async log purge

2022-10-23 Thread GitBox
HeartSaVioR closed pull request #38313: [SPARK-40849][SS] Async log purge URL: https://github.com/apache/spark/pull/38313 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] HeartSaVioR commented on pull request #38313: [SPARK-40849][SS] Async log purge

2022-10-23 Thread GitBox
HeartSaVioR commented on PR #38313: URL: https://github.com/apache/spark/pull/38313#issuecomment-1288310064 Thanks! Merging to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HeartSaVioR commented on pull request #38313: [SPARK-40849][SS] Async log purge

2022-10-23 Thread GitBox
HeartSaVioR commented on PR #38313: URL: https://github.com/apache/spark/pull/38313#issuecomment-1288309533 https://github.com/jerrypeng/spark/actions/runs/3293047872/jobs/5447874146 Remaining steps are unrelated to this PR - only license check which is respected in this PR. --

[GitHub] [spark] panbingkun commented on a diff in pull request #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes

2022-10-23 Thread GitBox
panbingkun commented on code in PR #38332: URL: https://github.com/apache/spark/pull/38332#discussion_r1002824150 ## core/src/main/resources/error/error-classes.json: ## @@ -143,6 +143,11 @@ "Offset expression must be a literal." ] }, +

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38317: [SPARK-40851][INFRA ][SQL][TESTS] Make GA run successfully with the latest Java 8/11/17

2022-10-23 Thread GitBox
HyukjinKwon commented on code in PR #38317: URL: https://github.com/apache/spark/pull/38317#discussion_r1002822821 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala: ## @@ -937,16 +937,16 @@ class DateTimeUtilsSuite extends SparkFunSuite

[GitHub] [spark] zhengruifeng commented on pull request #38346: [SPARK-40880][SQL] Reimplement `summary` with dataframe operations

2022-10-23 Thread GitBox
zhengruifeng commented on PR #38346: URL: https://github.com/apache/spark/pull/38346#issuecomment-1288303733 thank you @HyukjinKwon ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon closed pull request #38346: [SPARK-40880][SQL] Reimplement `summary` with dataframe operations

2022-10-23 Thread GitBox
HyukjinKwon closed pull request #38346: [SPARK-40880][SQL] Reimplement `summary` with dataframe operations URL: https://github.com/apache/spark/pull/38346 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] HyukjinKwon commented on pull request #38346: [SPARK-40880][SQL] Reimplement `summary` with dataframe operations

2022-10-23 Thread GitBox
HyukjinKwon commented on PR #38346: URL: https://github.com/apache/spark/pull/38346#issuecomment-1288302963 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #38354: [SPARK-40882][INFRA] Upgrade actions/setup-java to v3 with distribution specified

2022-10-23 Thread GitBox
HyukjinKwon commented on code in PR #38354: URL: https://github.com/apache/spark/pull/38354#discussion_r1002821239 ## .github/workflows/build_and_test.yml: ## @@ -227,8 +227,9 @@ jobs: restore-keys: | ${{ matrix.java }}-${{ matrix.hadoop }}-coursier- -

[GitHub] [spark] zhengruifeng commented on pull request #38340: [SPARK-40877][SQL] Reimplement `crosstab` with dataframe operations

2022-10-23 Thread GitBox
zhengruifeng commented on PR #38340: URL: https://github.com/apache/spark/pull/38340#issuecomment-1288300559 thank you @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] Yikun commented on a diff in pull request #38354: [SPARK-40882][INFRA] Upgrade actions/setup-java to v3 with distribution specified

2022-10-23 Thread GitBox
Yikun commented on code in PR #38354: URL: https://github.com/apache/spark/pull/38354#discussion_r1002820197 ## .github/workflows/build_and_test.yml: ## @@ -227,8 +227,9 @@ jobs: restore-keys: | ${{ matrix.java }}-${{ matrix.hadoop }}-coursier- - name:

[GitHub] [spark] Yikun commented on a diff in pull request #38354: [SPARK-40882][INFRA] Upgrade actions/setup-java to v3 with distribution specified

2022-10-23 Thread GitBox
Yikun commented on code in PR #38354: URL: https://github.com/apache/spark/pull/38354#discussion_r1002820197 ## .github/workflows/build_and_test.yml: ## @@ -227,8 +227,9 @@ jobs: restore-keys: | ${{ matrix.java }}-${{ matrix.hadoop }}-coursier- - name:

[GitHub] [spark] Yikun commented on pull request #38353: [SPARK-40881][INFRA] Upgrade actions/cache to v3 and actions/upload-artifact to v3

2022-10-23 Thread GitBox
Yikun commented on PR #38353: URL: https://github.com/apache/spark/pull/38353#issuecomment-1288297240 @srowen Thanks, might be rebase issue. Let me do a rebase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] yaooqinn commented on pull request #38032: [WIP][SPARK-40597][CORE] local mode should respect TASK_MAX_FAILURES like all other cluster managers

2022-10-23 Thread GitBox
yaooqinn commented on PR #38032: URL: https://github.com/apache/spark/pull/38032#issuecomment-1288296994 > What is the scenario where we are looking for this change to be useful ? accessing distributed data from distribute storage w/ local mode. -- This is an automated message

[GitHub] [spark] HyukjinKwon closed pull request #38340: [SPARK-40877][SQL] Reimplement `crosstab` with dataframe operations

2022-10-23 Thread GitBox
HyukjinKwon closed pull request #38340: [SPARK-40877][SQL] Reimplement `crosstab` with dataframe operations URL: https://github.com/apache/spark/pull/38340 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] HyukjinKwon commented on pull request #38340: [SPARK-40877][SQL] Reimplement `crosstab` with dataframe operations

2022-10-23 Thread GitBox
HyukjinKwon commented on PR #38340: URL: https://github.com/apache/spark/pull/38340#issuecomment-1288287684 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

  1   2   >