[GitHub] [spark] pralabhkumar commented on pull request #35191: [SPARK-37491][PYTHON]Fix Series.asof for unsorted values

pralabhkumar commented on pull request #35191: URL: https://github.com/apache/spark/pull/35191#issuecomment-1017182630 @HyukjinKwon Thx for the comments , working on it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] yutoacts commented on pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

yutoacts commented on pull request #35239: URL: https://github.com/apache/spark/pull/35239#issuecomment-1017181212 Also moved `ALTER OR CHANGE COLUMNS` example a bit above to match the order of the documentation. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] yutoacts commented on a change in pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

yutoacts commented on a change in pull request #35239: URL: https://github.com/apache/spark/pull/35239#discussion_r788421829 ## File path: docs/sql-ref-syntax-ddl-alter-table.md ## @@ -225,6 +294,25 @@ ALTER TABLE table_identifier [ partition_spec ] SET LOCATION

[GitHub] [spark] yutoacts commented on pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

yutoacts commented on pull request #35239: URL: https://github.com/apache/spark/pull/35239#issuecomment-1017178500 @huaxingao thanks for the review. just added examples. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] AngersZhuuuu commented on pull request #35254: [SPARK-37966][SQL] Static partition insert should write _SUCCESS under partition location

AngersZh commented on pull request #35254: URL: https://github.com/apache/spark/pull/35254#issuecomment-1017167931 ping @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] otterc commented on a change in pull request #35085: [SPARK-37618][CORE] Remove shuffle blocks using the shuffle service for released executors

otterc commented on a change in pull request #35085: URL: https://github.com/apache/spark/pull/35085#discussion_r788396539 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -2742,6 +2743,16 @@ private[spark] object Utils extends Logging { new

[GitHub] [spark] otterc commented on a change in pull request #35085: [SPARK-37618][CORE] Remove shuffle blocks using the shuffle service for released executors

otterc commented on a change in pull request #35085: URL: https://github.com/apache/spark/pull/35085#discussion_r788396539 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -2742,6 +2743,16 @@ private[spark] object Utils extends Logging { new

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #35254: [SPARK-37966][SQL] Static partition insert should write _SUCCESS under partition location

AngersZh commented on a change in pull request #35254: URL: https://github.com/apache/spark/pull/35254#discussion_r788393183 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala ## @@ -105,10 +105,22 @@

[GitHub] [spark] itholic commented on a change in pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

itholic commented on a change in pull request #35239: URL: https://github.com/apache/spark/pull/35239#discussion_r788387632 ## File path: docs/sql-ref-syntax-ddl-alter-table.md ## @@ -225,6 +294,25 @@ ALTER TABLE table_identifier [ partition_spec ] SET LOCATION 'new_location'

[GitHub] [spark] itholic commented on a change in pull request #35191: [SPARK-37491][PYTHON]Fix Series.asof for unsorted values

itholic commented on a change in pull request #35191: URL: https://github.com/apache/spark/pull/35191#discussion_r788379590 ## File path: python/pyspark/pandas/series.py ## @@ -5228,22 +5228,62 @@ def asof(self, where: Union[Any, List]) -> Union[Scalar, "Series"]:

[GitHub] [spark] itholic commented on a change in pull request #35191: [SPARK-37491][PYTHON]Fix Series.asof for unsorted values

itholic commented on a change in pull request #35191: URL: https://github.com/apache/spark/pull/35191#discussion_r788362544 ## File path: python/pyspark/pandas/series.py ## @@ -5228,22 +5228,62 @@ def asof(self, where: Union[Any, List]) -> Union[Scalar, "Series"]:

[GitHub] [spark] beliefer commented on pull request #35241: [SPARK-37953][SQL] Improve the implement of `UnevaluableAggregate`

beliefer commented on pull request #35241: URL: https://github.com/apache/spark/pull/35241#issuecomment-1017149787 ping @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #35207: [SPARK-37907][SQL] InvokeLike support ConstantFolding

AngersZh commented on a change in pull request #35207: URL: https://github.com/apache/spark/pull/35207#discussion_r788358766 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala ## @@ -50,6 +50,8 @@ trait InvokeLike

[GitHub] [spark] pralabhkumar commented on pull request #35191: [SPARK-37491][PYTHON]Fix Series.asof for unsorted values

pralabhkumar commented on pull request #35191: URL: https://github.com/apache/spark/pull/35191#issuecomment-1017103938 @itholic , Please let me know about your review comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan closed pull request #35130: [SPARK-37839][SQL] DS V2 supports partial aggregate push-down `AVG`

cloud-fan closed pull request #35130: URL: https://github.com/apache/spark/pull/35130 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] cloud-fan commented on pull request #35130: [SPARK-37839][SQL] DS V2 supports partial aggregate push-down `AVG`

cloud-fan commented on pull request #35130: URL: https://github.com/apache/spark/pull/35130#issuecomment-1017102913 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] gengliangwang commented on a change in pull request #35251: [SPARK-37963][SQL] Need to update Partition URI after renaming table in InMemoryCatalog

gengliangwang commented on a change in pull request #35251: URL: https://github.com/apache/spark/pull/35251#discussion_r788336316 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableRenameSuiteBase.scala ## @@ -126,4 +126,14 @@ trait

[GitHub] [spark] dchvn commented on pull request #34363: [SPARK-37083][PYTHON] Inline type hints for python/pyspark/accumulators.py

dchvn commented on pull request #34363: URL: https://github.com/apache/spark/pull/34363#issuecomment-1017093623 > > sorry I completely missed the context here. What's the status now? > > So far, there is no response to issue on mypy tracker, but we can be fairly sure that the issue

[GitHub] [spark] AngersZhuuuu opened a new pull request #35254: [SPARK-37966][SQL] Static partition insert should write _SUCCESS under partition location

AngersZh opened a new pull request #35254: URL: https://github.com/apache/spark/pull/35254 ### What changes were proposed in this pull request? Currently, when we use static partition insert to write a partition with data source API, the `_SUCCESS` flag file is write under table

[GitHub] [spark] beliefer commented on a change in pull request #35166: [SPARK-37867][SQL] Compile aggregate functions of build-in JDBC dialect

beliefer commented on a change in pull request #35166: URL: https://github.com/apache/spark/pull/35166#discussion_r788318836 ## File path: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCTest.scala ## @@ -348,4 +354,145 @@ private[v2] trait

[GitHub] [spark] AngersZhuuuu opened a new pull request #35253: [WIP][SPARK-37965][SQL]Remove check field name when reading/writing existing data in Orc

AngersZh opened a new pull request #35253: URL: https://github.com/apache/spark/pull/35253 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was

[GitHub] [spark] beliefer commented on a change in pull request #35060: [SPARK-28137][SQL] Data Type Formatting Functions: `to_number`

beliefer commented on a change in pull request #35060: URL: https://github.com/apache/spark/pull/35060#discussion_r788313185 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/NumberFormatterSuite.scala ## @@ -19,76 +19,75 @@ package

[GitHub] [spark] AngersZhuuuu commented on pull request #35229: [SPARK-27442][SQL] Remove check field name when reading/writing data in parquet

AngersZh commented on pull request #35229: URL: https://github.com/apache/spark/pull/35229#issuecomment-1017074485 ping @cloud-fan All related test removed. For check supported field name, remained test in `avro` module. -- This is an automated message from the Apache Git Service.

[GitHub] [spark] beliefer commented on a change in pull request #35060: [SPARK-28137][SQL] Data Type Formatting Functions: `to_number`

beliefer commented on a change in pull request #35060: URL: https://github.com/apache/spark/pull/35060#discussion_r788313185 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/NumberFormatterSuite.scala ## @@ -19,76 +19,75 @@ package

[GitHub] [spark] itholic commented on a change in pull request #35082: [SPARK-37677][PYTHON] Decompress the ZIP file and grant the executable permission to the file

itholic commented on a change in pull request #35082: URL: https://github.com/apache/spark/pull/35082#discussion_r788308585 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -605,6 +606,18 @@ private[spark] object Utils extends Logging { } } +

[GitHub] [spark] LuciferYang commented on pull request #35226: [SPARK-37928][SQL][TESTS] Add Parquet Data Page V2 test scenario to `DataSourceReadBenchmark`

LuciferYang commented on pull request #35226: URL: https://github.com/apache/spark/pull/35226#issuecomment-1017065796 thanks all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] cloud-fan commented on a change in pull request #35221: [SPARK-37923][SQL] Generate partition transforms for BucketSpec inside parser

cloud-fan commented on a change in pull request #35221: URL: https://github.com/apache/spark/pull/35221#discussion_r788305390 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Implicits.scala ## @@ -49,21 +51,27 @@ private[sql] object

[GitHub] [spark] cloud-fan commented on a change in pull request #35166: [SPARK-37867][SQL] Compile aggregate functions of build-in JDBC dialect

cloud-fan commented on a change in pull request #35166: URL: https://github.com/apache/spark/pull/35166#discussion_r788300433 ## File path: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCTest.scala ## @@ -348,4 +354,145 @@ private[v2]

[GitHub] [spark] AngersZhuuuu commented on pull request #35207: [SPARK-37907][SQL] InvokeLike support ConstantFolding

AngersZh commented on pull request #35207: URL: https://github.com/apache/spark/pull/35207#issuecomment-1017056154 since #35243 merged, we can start review this one. Any more suggestion? -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] venkata91 commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

venkata91 commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r788292151 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -1364,6 +1430,7 @@ private[spark] class MapOutputTrackerWorker(conf:

[GitHub] [spark] venkata91 commented on pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

venkata91 commented on pull request #34122: URL: https://github.com/apache/spark/pull/34122#issuecomment-1017042155 Addressed the review comments @mridulm Please take a look whenever you have some time. Thanks! -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] venkata91 commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

venkata91 commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r788287896 ## File path: core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala ## @@ -910,4 +910,32 @@ class MapOutputTrackerSuite extends

[GitHub] [spark] venkata91 commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

venkata91 commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r788287411 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -1364,6 +1430,7 @@ private[spark] class MapOutputTrackerWorker(conf:

[GitHub] [spark] venkata91 commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

venkata91 commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r788286262 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -1176,6 +1223,9 @@ private[spark] class MapOutputTrackerWorker(conf:

[GitHub] [spark] venkata91 commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

venkata91 commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r788286262 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -1176,6 +1223,9 @@ private[spark] class MapOutputTrackerWorker(conf:

[GitHub] [spark] venkata91 commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

venkata91 commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r788286262 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -1176,6 +1223,9 @@ private[spark] class MapOutputTrackerWorker(conf:

[GitHub] [spark] venkata91 commented on a change in pull request #34122: [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle

venkata91 commented on a change in pull request #34122: URL: https://github.com/apache/spark/pull/34122#discussion_r788286262 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -1176,6 +1223,9 @@ private[spark] class MapOutputTrackerWorker(conf:

[GitHub] [spark] HyukjinKwon commented on pull request #35187: [SPARK-37891][CORE] Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global

HyukjinKwon commented on pull request #35187: URL: https://github.com/apache/spark/pull/35187#issuecomment-1017026560 @tianhanhu would need to rebase again .. :-). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] zhengruifeng commented on pull request #35247: [SPARK-37959][ML] Fix the UT of checking norm in KMeans & BiKMeans

zhengruifeng commented on pull request #35247: URL: https://github.com/apache/spark/pull/35247#issuecomment-1017022562 thank you all! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] tianhanhu commented on pull request #35187: [SPARK-37891][CORE] Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global

tianhanhu commented on pull request #35187: URL: https://github.com/apache/spark/pull/35187#issuecomment-1017014826 Just re-enabled workflow. Thanks for the catch @HyukjinKwon :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] beliefer commented on a change in pull request #35166: [SPARK-37867][SQL] Compile aggregate functions of build-in JDBC dialect

beliefer commented on a change in pull request #35166: URL: https://github.com/apache/spark/pull/35166#discussion_r788268740 ## File path: sql/core/src/main/scala/org/apache/spark/sql/jdbc/DB2Dialect.scala ## @@ -20,13 +20,26 @@ package org.apache.spark.sql.jdbc import

[GitHub] [spark] HyukjinKwon edited a comment on pull request #35187: [SPARK-37891][CORE] Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global

HyukjinKwon edited a comment on pull request #35187: URL: https://github.com/apache/spark/pull/35187#issuecomment-1016993279 Oh I see. looks like the main build job got disabled (https://github.com/tianhanhu/spark/actions/workflows/build_and_test.yml): ![Screen Shot 2022-01-20 at 9 24

[GitHub] [spark] HyukjinKwon commented on pull request #35187: [SPARK-37891][CORE] Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global

HyukjinKwon commented on pull request #35187: URL: https://github.com/apache/spark/pull/35187#issuecomment-1016993279 Oh I see. looks like the main build job got disabled: ![Screen Shot 2022-01-20 at 9 24 37

[GitHub] [spark] tianhanhu commented on pull request #35187: [SPARK-37891][CORE] Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global

tianhanhu commented on pull request #35187: URL: https://github.com/apache/spark/pull/35187#issuecomment-1016991991 @HyukjinKwon I do have actions all allowed for my forked Spark, and I just did the rebase... -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] HyukjinKwon commented on a change in pull request #35251: [SPARK-37963][SQL] Need to update Partition URI after renaming table in InMemoryCatalog

HyukjinKwon commented on a change in pull request #35251: URL: https://github.com/apache/spark/pull/35251#discussion_r788251304 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableRenameSuiteBase.scala ## @@ -126,4 +126,14 @@ trait

[GitHub] [spark] HyukjinKwon commented on pull request #35250: [SPARK-37961][SQL] Override maxRows/maxRowsPerPartition for some logical operators

HyukjinKwon commented on pull request #35250: URL: https://github.com/apache/spark/pull/35250#issuecomment-1016989621 cc @wangyum FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon commented on pull request #35187: [SPARK-37891][CORE] Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global

HyukjinKwon commented on pull request #35187: URL: https://github.com/apache/spark/pull/35187#issuecomment-1016988272 Thanks for cc'ing me @jiangxb1987. @tianhanhu, Jenkins reached EOL so Apache Spark currently relies on GitHub Actions. Due to the lack of resources allocated to Apache

[GitHub] [spark] github-actions[bot] commented on pull request #34223: [SPARK-36957][SQL] Add Aggregate function Product to SQL function

github-actions[bot] commented on pull request #34223: URL: https://github.com/apache/spark/pull/34223#issuecomment-1016987279 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue

[GitHub] [spark] github-actions[bot] closed pull request #34198: [SPARK-36300][SQL] Refactor eleventh set of 20 in QueryExecutionErrors to use error classes

github-actions[bot] closed pull request #34198: URL: https://github.com/apache/spark/pull/34198 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] github-actions[bot] commented on pull request #33550: [SPARK-36321][K8S] Do not fail application in kubernetes if name is too long

github-actions[bot] commented on pull request #33550: URL: https://github.com/apache/spark/pull/33550#issuecomment-1016987308 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue

[GitHub] [spark] jiangxb1987 commented on pull request #35187: [SPARK-37891][CORE] Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global

jiangxb1987 commented on pull request #35187: URL: https://github.com/apache/spark/pull/35187#issuecomment-1016984285 cc @HyukjinKwon can you help determine why the build workflow failed? -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] dongjoon-hyun commented on pull request #35243: [SPARK-37957][SQL] Correctly pass deterministic flag for V2 scalar functions

dongjoon-hyun commented on pull request #35243: URL: https://github.com/apache/spark/pull/35243#issuecomment-1016979317 Thank you all! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] gatorsmile removed a comment on pull request #35187: [SPARK-37891][CORE] Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global

gatorsmile removed a comment on pull request #35187: URL: https://github.com/apache/spark/pull/35187#issuecomment-1016967348 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] gatorsmile commented on pull request #35187: [SPARK-37891][CORE] Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global

gatorsmile commented on pull request #35187: URL: https://github.com/apache/spark/pull/35187#issuecomment-1016967348 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] huaxingao commented on pull request #35239: [SPARK-37952][DOCS] Add missing statements to ALTER TABLE document

huaxingao commented on pull request #35239: URL: https://github.com/apache/spark/pull/35239#issuecomment-1016965975 @yutoacts Thanks for working on this! Could you please add some examples? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] zero323 commented on a change in pull request #34293: [SPARK-37014][PYTHON] Inline type hints for python/pyspark/streaming/context.py

zero323 commented on a change in pull request #34293: URL: https://github.com/apache/spark/pull/34293#discussion_r788224761 ## File path: python/pyspark/streaming/context.py ## @@ -264,7 +280,9 @@ def checkpoint(self, directory): """

[GitHub] [spark] otterc commented on pull request #34934: [SPARK-37675][CORE][SHUFFLE] Return PushMergedRemoteMetaFailedFetchResult if no available push-merged block

otterc commented on pull request #34934: URL: https://github.com/apache/spark/pull/34934#issuecomment-1016956929 > @otterc I reproduced this issue today and sent a email to you with logs and spark confs you requested

[GitHub] [spark] zero323 commented on a change in pull request #34293: [SPARK-37014][PYTHON] Inline type hints for python/pyspark/streaming/context.py

zero323 commented on a change in pull request #34293: URL: https://github.com/apache/spark/pull/34293#discussion_r788199973 ## File path: python/pyspark/streaming/context.py ## @@ -264,7 +280,9 @@ def checkpoint(self, directory): """

[GitHub] [spark] tianhanhu commented on pull request #35187: [SPARK-37891][CORE] Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global

tianhanhu commented on pull request #35187: URL: https://github.com/apache/spark/pull/35187#issuecomment-1016934693 Jenkins test this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] zero323 commented on a change in pull request #34293: [SPARK-37014][PYTHON] Inline type hints for python/pyspark/streaming/context.py

zero323 commented on a change in pull request #34293: URL: https://github.com/apache/spark/pull/34293#discussion_r788199973 ## File path: python/pyspark/streaming/context.py ## @@ -264,7 +280,9 @@ def checkpoint(self, directory): """

[GitHub] [spark] zero323 commented on a change in pull request #34324: [SPARK-37015][PYTHON] Inline type hints for python/pyspark/streaming/dstream.py

zero323 commented on a change in pull request #34324: URL: https://github.com/apache/spark/pull/34324#discussion_r788198862 ## File path: python/pyspark/streaming/dstream.py ## @@ -51,122 +76,165 @@ class DStream(object): - A function that is used to generate an RDD

[GitHub] [spark] zero323 commented on pull request #34363: [SPARK-37083][PYTHON] Inline type hints for python/pyspark/accumulators.py

zero323 commented on pull request #34363: URL: https://github.com/apache/spark/pull/34363#issuecomment-1016925797 > sorry I completely missed the context here. What's the status now? So far, there is no response to issue on mypy tracker, but we can be fairly sure that the issue that

[GitHub] [spark] sunchao commented on pull request #35243: [SPARK-37957][SQL] Correctly pass deterministic flag for V2 scalar functions

sunchao commented on pull request #35243: URL: https://github.com/apache/spark/pull/35243#issuecomment-1016918767 Thanks! merged to master and branch-3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] sunchao closed pull request #35243: [SPARK-37957][SQL] Correctly pass deterministic flag for V2 scalar functions

sunchao closed pull request #35243: URL: https://github.com/apache/spark/pull/35243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] c21 commented on a change in pull request #35220: [SPARK-37922][SQL] Combine to one cast if we can safely up-cast two casts

c21 commented on a change in pull request #35220: URL: https://github.com/apache/spark/pull/35220#discussion_r788137705 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala ## @@ -1037,13 +1037,26 @@ object SimplifyCasts extends

[GitHub] [spark] c21 commented on a change in pull request #35220: [SPARK-37922][SQL] Combine to one cast if we can safely up-cast two casts

c21 commented on a change in pull request #35220: URL: https://github.com/apache/spark/pull/35220#discussion_r788134305 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala ## @@ -1037,13 +1037,26 @@ object SimplifyCasts extends

[GitHub] [spark] zero323 opened a new pull request #35252: [WIP][SPARK-37154][PYTHON] Inline hints for pyspark.rdd.

zero323 opened a new pull request #35252: URL: https://github.com/apache/spark/pull/35252 ### What changes were proposed in this pull request? This PR proposes migration of type hints for `pyspark.rdd` from stub file to inline annotation. ### Why are the changes

[GitHub] [spark] mridulm commented on a change in pull request #35085: [SPARK-37618][CORE] Remove shuffle blocks using the shuffle service for released executors

mridulm commented on a change in pull request #35085: URL: https://github.com/apache/spark/pull/35085#discussion_r788095136 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -2742,6 +2743,16 @@ private[spark] object Utils extends Logging { new

[GitHub] [spark] sunchao commented on a change in pull request #35243: [SPARK-37957][SQL] Correctly pass deterministic flag for V2 scalar functions

sunchao commented on a change in pull request #35243: URL: https://github.com/apache/spark/pull/35243#discussion_r788040266 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala ## @@ -259,6 +262,7 @@ case class

[GitHub] [spark] srowen commented on pull request #35230: [SPARK-37934] [Build] Upgrade Jetty version to 9.4.44

srowen commented on pull request #35230: URL: https://github.com/apache/spark/pull/35230#issuecomment-1016754449 Possible, how much does this affect spark users? Generally wouldn't back port anything but non trivial bug fixes -- This is an automated message from the Apache Git Service.

[GitHub] [spark] this commented on pull request #35230: [SPARK-37934] [Build] Upgrade Jetty version to 9.4.44

this commented on pull request #35230: URL: https://github.com/apache/spark/pull/35230#issuecomment-1016752389 @srowen Do we need to port this to the `branch-3.2`branch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] srowen closed pull request #35230: [SPARK-37934] [Build] Upgrade Jetty version to 9.4.44

srowen closed pull request #35230: URL: https://github.com/apache/spark/pull/35230 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] srowen commented on pull request #35230: [SPARK-37934] [Build] Upgrade Jetty version to 9.4.44

srowen commented on pull request #35230: URL: https://github.com/apache/spark/pull/35230#issuecomment-1016738904 Yep all green now. I will merge to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] cloud-fan commented on a change in pull request #35243: [SPARK-37957][SQL] Correctly pass deterministic flag for V2 scalar functions

cloud-fan commented on a change in pull request #35243: URL: https://github.com/apache/spark/pull/35243#discussion_r788007794 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala ## @@ -259,6 +262,7 @@ case class

[GitHub] [spark] sunchao closed pull request #35212: [SPARK-36879][SQL][FOLLOWUP] Address comments and fix code style

sunchao closed pull request #35212: URL: https://github.com/apache/spark/pull/35212 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] sunchao commented on pull request #35212: [SPARK-36879][SQL][FOLLOWUP] Address comments and fix code style

sunchao commented on pull request #35212: URL: https://github.com/apache/spark/pull/35212#issuecomment-1016717310 Thanks @parthchandra , merged to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] parthchandra commented on pull request #35212: [SPARK-36879][SQL][FOLLOWUP] Support Parquet v2 data page encodings for the vectorized path

parthchandra commented on pull request #35212: URL: https://github.com/apache/spark/pull/35212#issuecomment-1016709661 @viirya, @LuciferYang Updated the description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] sunchao commented on pull request #35226: [SPARK-37928][SQL][TESTS] Add Parquet Data Page V2 test scenario to `DataSourceReadBenchmark`

sunchao commented on pull request #35226: URL: https://github.com/apache/spark/pull/35226#issuecomment-1016703139 Thanks @LuciferYang , merged to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] sunchao closed pull request #35226: [SPARK-37928][SQL][TESTS] Add Parquet Data Page V2 test scenario to `DataSourceReadBenchmark`

sunchao closed pull request #35226: URL: https://github.com/apache/spark/pull/35226 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] sunchao commented on a change in pull request #35243: [SPARK-37957][SQL] Correctly pass deterministic flag for V2 scalar functions

sunchao commented on a change in pull request #35243: URL: https://github.com/apache/spark/pull/35243#discussion_r787978011 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala ## @@ -259,6 +262,7 @@ case class

[GitHub] [spark] huaxingao commented on pull request #35247: [SPARK-37959][ML] Fix the UT of checking norm in KMeans & BiKMeans

huaxingao commented on pull request #35247: URL: https://github.com/apache/spark/pull/35247#issuecomment-1016689260 Merged to master/3.2. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] huaxingao closed pull request #35247: [SPARK-37959][ML] Fix the UT of checking norm in KMeans & BiKMeans

huaxingao closed pull request #35247: URL: https://github.com/apache/spark/pull/35247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] this commented on pull request #35230: [SPARK-37934] [Build] Upgrade Jetty version to 9.4.44

this commented on pull request #35230: URL: https://github.com/apache/spark/pull/35230#issuecomment-1016685667 My 3rd attempted passed https://github.com/this/spark/runs/4869742406?check_suite_focus=true. However that is not reflected here in the checks list. Do I need to push an empty

[GitHub] [spark] vinooganesh closed pull request #35224: [SPARK-32165][SQL] Ensure Spark only initiates SharedState once across SparkSessions

vinooganesh closed pull request #35224: URL: https://github.com/apache/spark/pull/35224 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] vinooganesh commented on pull request #35224: [SPARK-32165][SQL] Ensure Spark only initiates SharedState once across SparkSessions

vinooganesh commented on pull request #35224: URL: https://github.com/apache/spark/pull/35224#issuecomment-1016644297 Closing per comment in https://issues.apache.org/jira/browse/SPARK-32165. Thanks @cloud-fan ! -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] Yikun edited a comment on pull request #35236: [SPARK-37903][PYTHON][FOLLOW-UP] Raise TypeError with no return function

Yikun edited a comment on pull request #35236: URL: https://github.com/apache/spark/pull/35236#issuecomment-1016123760 Thanks, remember to also update the PR description/title. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] dnskr commented on pull request #35224: [SPARK-32165][SQL] Ensure Spark only initiates SharedState once across SparkSessions

dnskr commented on pull request #35224: URL: https://github.com/apache/spark/pull/35224#issuecomment-1016630939 @vinooganesh Related jira ticket https://issues.apache.org/jira/browse/SPARK-32165 has been closed so you can close the PR. -- This is an automated message from the Apache

[GitHub] [spark] dnskr closed pull request #35234: [SPARK-32165][SQL] SessionState leaks SparkListener with multiple SparkSession

dnskr closed pull request #35234: URL: https://github.com/apache/spark/pull/35234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] AmplabJenkins commented on pull request #35244: [SPARK-37956][DOCS] Add Python and Java examples of Parquet encryption in Spark SQL to documentation

AmplabJenkins commented on pull request #35244: URL: https://github.com/apache/spark/pull/35244#issuecomment-1016625844 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] cloud-fan commented on pull request #35168: [SPARK-37865][SQL]Spark should not dedup the grouping Expressions when the first child of union has duplicate columns

cloud-fan commented on pull request #35168: URL: https://github.com/apache/spark/pull/35168#issuecomment-1016585193 I think a complete fix is to let `Union` has fresh attribute IDs for its output, as technically `Union`'s outputs are totally different from its first child's outputs. It's

[GitHub] [spark] Kimahriman commented on a change in pull request #35085: [SPARK-37618][CORE] Remove shuffle blocks using the shuffle service for released executors

Kimahriman commented on a change in pull request #35085: URL: https://github.com/apache/spark/pull/35085#discussion_r787862613 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -2742,6 +2743,16 @@ private[spark] object Utils extends Logging { new

[GitHub] [spark] gengliangwang commented on a change in pull request #35251: [SPARK-37963][SQL] Need to update Partition URI after renaming table in InMemoryCatalog

gengliangwang commented on a change in pull request #35251: URL: https://github.com/apache/spark/pull/35251#discussion_r787850280 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableRenameSuite.scala ## @@ -74,6 +74,16 @@ trait

[GitHub] [spark] cloud-fan commented on a change in pull request #35166: [SPARK-37867][SQL] Compile aggregate functions of build-in JDBC dialect

cloud-fan commented on a change in pull request #35166: URL: https://github.com/apache/spark/pull/35166#discussion_r787841166 ## File path: sql/core/src/main/scala/org/apache/spark/sql/jdbc/DB2Dialect.scala ## @@ -20,13 +20,26 @@ package org.apache.spark.sql.jdbc import

[GitHub] [spark] pan3793 commented on pull request #35223: [SPARK-37925][DOC] Update document to mention the workaround for YARN-11053

pan3793 commented on pull request #35223: URL: https://github.com/apache/spark/pull/35223#issuecomment-1016547917 @itholic could you please take a look again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] srowen commented on pull request #35247: [SPARK-37959][ML] Fix the UT of checking norm in KMeans & BiKMeans

srowen commented on pull request #35247: URL: https://github.com/apache/spark/pull/35247#issuecomment-1016544490 Oh right, I'm not reading. OK yes delete them -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] zhengruifeng commented on pull request #35247: [SPARK-37959][ML] Fix the UT of checking norm in KMeans & BiKMeans

zhengruifeng commented on pull request #35247: URL: https://github.com/apache/spark/pull/35247#issuecomment-1016542650 Euclidean distance doesn't have this attribute. > then I wonder how that test ever passed they were not assertions, just expressions. -- This is an

[GitHub] [spark] srowen commented on pull request #35247: [SPARK-37959][ML] Fix the UT of checking norm in KMeans & BiKMeans

srowen commented on pull request #35247: URL: https://github.com/apache/spark/pull/35247#issuecomment-1016527278 OK but not for Euclidean distance? then I wonder how that test ever passed, unless the data just happens to produce centers at distance 1 from the origin -- This is an

[GitHub] [spark] cloud-fan commented on a change in pull request #35060: [SPARK-28137][SQL] Data Type Formatting Functions: `to_number`

cloud-fan commented on a change in pull request #35060: URL: https://github.com/apache/spark/pull/35060#discussion_r787795300 ## File path: sql/core/src/test/resources/sql-tests/inputs/string-functions.sql ## @@ -124,4 +124,22 @@ SELECT endswith('Spark SQL', 'QL'); SELECT

[GitHub] [spark] cloud-fan commented on a change in pull request #35060: [SPARK-28137][SQL] Data Type Formatting Functions: `to_number`

cloud-fan commented on a change in pull request #35060: URL: https://github.com/apache/spark/pull/35060#discussion_r787795046 ## File path: sql/core/src/test/resources/sql-tests/inputs/string-functions.sql ## @@ -124,4 +124,22 @@ SELECT endswith('Spark SQL', 'QL'); SELECT

[GitHub] [spark] cloud-fan commented on a change in pull request #35060: [SPARK-28137][SQL] Data Type Formatting Functions: `to_number`

cloud-fan commented on a change in pull request #35060: URL: https://github.com/apache/spark/pull/35060#discussion_r787793732 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/NumberFormatterSuite.scala ## @@ -19,76 +19,75 @@ package

  1   2   >