[GitHub] [spark] SparkQA removed a comment on pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

SparkQA removed a comment on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-871794553 **[Test build #140469 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140469/testReport)** for PR 33160 at commit [`0bf197b`](https://gi

[GitHub] [spark] SparkQA commented on pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

SparkQA commented on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-871855243 **[Test build #140469 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140469/testReport)** for PR 33160 at commit [`0bf197b`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #33162: [WIP][SPARK-35615] Make all basic operators data-type-based

SparkQA removed a comment on pull request #33162: URL: https://github.com/apache/spark/pull/33162#issuecomment-871843521 **[Test build #140472 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140472/testReport)** for PR 33162 at commit [`39d4120`](https://gi

[GitHub] [spark] mridulm edited a comment on pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a

mridulm edited a comment on pull request #33078: URL: https://github.com/apache/spark/pull/33078#issuecomment-871854550 @Ngone51 I am leaving on slightly long-ish vacation without access to my laptop .. will it be possible for you to shepard this pr and #33034 ? These are the last two pe

[GitHub] [spark] SparkQA commented on pull request #33162: [WIP][SPARK-35615] Make all basic operators data-type-based

SparkQA commented on pull request #33162: URL: https://github.com/apache/spark/pull/33162#issuecomment-871854756 **[Test build #140472 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140472/testReport)** for PR 33162 at commit [`39d4120`](https://github.co

[GitHub] [spark] mridulm commented on pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better

mridulm commented on pull request #33078: URL: https://github.com/apache/spark/pull/33078#issuecomment-871854550 @Ngone51 I am leaving on slightly long-ish vacation without access to my laptop .. will it be possible for you to shepard this pr and #33034 ? These are the last two pending f

[GitHub] [spark] cloud-fan commented on a change in pull request #33142: [SPARK-35940][SQL] Refactor EquivalentExpressions to make it more efficient

cloud-fan commented on a change in pull request #33142: URL: https://github.com/apache/spark/pull/33142#discussion_r661921386 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala ## @@ -170,65 +171,75 @@ class Equivale

[GitHub] [spark] mridulm commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

mridulm commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661786876 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -403,38 +394,78 @@ public MergeSta

[GitHub] [spark] kotlovs commented on pull request #33154: [SPARK-35949][CORE]Fixes bug for sparkContext stopped on client mode

kotlovs commented on pull request #33154: URL: https://github.com/apache/spark/pull/33154#issuecomment-871853541 @sunpe Could you please tell in more details, what is the problem with client mode? This code is called after exit from main() method, when the application is moving towards te

[GitHub] [spark] maropu commented on a change in pull request #33146: [SPARK-35912][SQL] Fix cast struct contains null value to string/struct

maropu commented on a change in pull request #33146: URL: https://github.com/apache/spark/pull/33146#discussion_r661919348 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala ## @@ -1136,4 +1136,51 @@ abstract class CastSuite

[GitHub] [spark] cloud-fan commented on a change in pull request #33142: [SPARK-35940][SQL] Refactor EquivalentExpressions to make it more efficient

cloud-fan commented on a change in pull request #33142: URL: https://github.com/apache/spark/pull/33142#discussion_r661919935 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala ## @@ -170,65 +171,75 @@ class Equivale

[GitHub] [spark] cloud-fan commented on a change in pull request #33142: [SPARK-35940][SQL] Refactor EquivalentExpressions to make it more efficient

cloud-fan commented on a change in pull request #33142: URL: https://github.com/apache/spark/pull/33142#discussion_r661919473 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala ## @@ -135,33 +125,47 @@ class Equivale

[GitHub] [spark] SparkQA commented on pull request #33038: [SPARK-35861][SS] Introduce "prefix match scan" feature on state store

SparkQA commented on pull request #33038: URL: https://github.com/apache/spark/pull/33038#issuecomment-871849227 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44984/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #31569: [SPARK-34443][CORE] Replace symbol literals with Symbol constructor invocations to comply with Scala 2.13

SparkQA commented on pull request #31569: URL: https://github.com/apache/spark/pull/31569#issuecomment-871846487 **[Test build #140475 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140475/testReport)** for PR 31569 at commit [`748bc9a`](https://github.com

[GitHub] [spark] maropu commented on a change in pull request #33142: [SPARK-35940][SQL] Refactor EquivalentExpressions to make it more efficient

maropu commented on a change in pull request #33142: URL: https://github.com/apache/spark/pull/33142#discussion_r661911895 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala ## @@ -170,65 +171,75 @@ class EquivalentE

[GitHub] [spark] SparkQA commented on pull request #33158: [SPARK-35888][SQL][FOLLOWUP] Return partition specs for all the shuffles

SparkQA commented on pull request #33158: URL: https://github.com/apache/spark/pull/33158#issuecomment-871845187 **[Test build #140474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140474/testReport)** for PR 33158 at commit [`b8a0362`](https://github.com

[GitHub] [spark] cloud-fan closed pull request #33158: [SPARK-35888][SQL][FOLLOWUP] Return partition specs for all the shuffles

cloud-fan closed pull request #33158: URL: https://github.com/apache/spark/pull/33158 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsu

[GitHub] [spark] cloud-fan commented on pull request #33158: [SPARK-35888][SQL][FOLLOWUP] Return partition specs for all the shuffles

cloud-fan commented on pull request #33158: URL: https://github.com/apache/spark/pull/33158#issuecomment-871844411 The last commit just fixes a typo, merging to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [spark] SparkQA commented on pull request #33028: [SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit

SparkQA commented on pull request #33028: URL: https://github.com/apache/spark/pull/33028#issuecomment-871843586 **[Test build #140473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140473/testReport)** for PR 33028 at commit [`2af82e5`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #33163: [SPARK-35960][BUILD][TEST] Bump the scalatest version to 3.2.9

SparkQA commented on pull request #33163: URL: https://github.com/apache/spark/pull/33163#issuecomment-871843529 **[Test build #140471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140471/testReport)** for PR 33163 at commit [`4a27d68`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #33162: [WIP][SPARK-35615] Make all basic operators data-type-based

SparkQA commented on pull request #33162: URL: https://github.com/apache/spark/pull/33162#issuecomment-871843521 **[Test build #140472 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140472/testReport)** for PR 33162 at commit [`39d4120`](https://github.com

[GitHub] [spark] ulysses-you commented on pull request #32883: [SPARK-35725][SQL] Support optimize skewed partitions in RebalancePartitions

ulysses-you commented on pull request #32883: URL: https://github.com/apache/spark/pull/32883#issuecomment-871841542 thank you all ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] holdenk opened a new pull request #33163: [SPARK-35960][BUILD][TEST] Bump the scalatest version to 3.2.9

holdenk opened a new pull request #33163: URL: https://github.com/apache/spark/pull/33163 ### What changes were proposed in this pull request? Bump the scalatest version to 3.2.9 ### Why are the changes needed? With the scalatestplus change to 3.2.9.0, recent sb

[GitHub] [spark] SparkQA commented on pull request #33038: [SPARK-35861][SS] Introduce "prefix match scan" feature on state store

SparkQA commented on pull request #33038: URL: https://github.com/apache/spark/pull/33038#issuecomment-871837962 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44984/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon edited a comment on pull request #33154: [SPARK-35949][CORE]Fixes bug for sparkContext stopped on client mode

HyukjinKwon edited a comment on pull request #33154: URL: https://github.com/apache/spark/pull/33154#issuecomment-871829941 cc @kotlovs, @dongjoon-hyun, @mridulm FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] HyukjinKwon commented on pull request #33154: [SPARK-35949][CORE]Fixes bug for sparkContext stopped on client mode

HyukjinKwon commented on pull request #33154: URL: https://github.com/apache/spark/pull/33154#issuecomment-871829941 cc @@kotlovs, @dongjoon-hyun, @mridulm FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [spark] xinrong-databricks opened a new pull request #33162: [WIP][SPARK-35615] Make all basic operators data-type-based

xinrong-databricks opened a new pull request #33162: URL: https://github.com/apache/spark/pull/33162 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

[GitHub] [spark] AmplabJenkins commented on pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

AmplabJenkins commented on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-871826207 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44983/ -- T

[GitHub] [spark] SparkQA commented on pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

SparkQA commented on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-871826187 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44983/ -- This is an automated message from the A

[GitHub] [spark] HeartSaVioR commented on pull request #32933: [SPARK-35785][SS] Cleanup support for RocksDB instance

HeartSaVioR commented on pull request #32933: URL: https://github.com/apache/spark/pull/32933#issuecomment-871824427 Could you please rebase this again? Looks like we have some "stale" changes here, like RocksDBLoader. You'll probably need to "cherry-pick" commits into the new branch

[GitHub] [spark] SparkQA commented on pull request #33038: [SPARK-35861][SS] Introduce "prefix match scan" feature on state store

SparkQA commented on pull request #33038: URL: https://github.com/apache/spark/pull/33038#issuecomment-871821101 **[Test build #140470 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140470/testReport)** for PR 33038 at commit [`36cdeab`](https://github.com

[GitHub] [spark] HyukjinKwon closed pull request #33159: [SPARK-35944][PYTHON] Introduce Name and Label type aliases

HyukjinKwon closed pull request #33159: URL: https://github.com/apache/spark/pull/33159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-un

[GitHub] [spark] HyukjinKwon commented on pull request #33159: [SPARK-35944][PYTHON] Introduce Name and Label type aliases

HyukjinKwon commented on pull request #33159: URL: https://github.com/apache/spark/pull/33159#issuecomment-871820630 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

AmplabJenkins removed a comment on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871818961 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44981/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

AmplabJenkins removed a comment on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871818959 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [spark] AmplabJenkins commented on pull request #33161: [WIP][SPARK-31973][SQL] Skip partial aggregates in run-time if reduction ratio is low

AmplabJenkins commented on pull request #33161: URL: https://github.com/apache/spark/pull/33161#issuecomment-871819110 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [spark] AmplabJenkins commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

AmplabJenkins commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871818959 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [spark] AmplabJenkins commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

AmplabJenkins commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871818961 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44981/ -- T

[GitHub] [spark] HeartSaVioR commented on a change in pull request #33038: [SPARK-35861][SS] Introduce "prefix match scan" feature on state store

HeartSaVioR commented on a change in pull request #33038: URL: https://github.com/apache/spark/pull/33038#discussion_r661894685 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala ## @@ -0,0 +1,172 @@ +/* + * Licen

[GitHub] [spark] shipra-a opened a new pull request #33161: [WIP][SPARK-31973][SQL] Skip partial aggregates in run-time if reduction ratio is low

shipra-a opened a new pull request #33161: URL: https://github.com/apache/spark/pull/33161 ### What changes were proposed in this pull request? This PR builds on top of https://github.com/apache/spark/pull/28804. In addition to the other PR, one other change is that the partial aggre

[GitHub] [spark] HyukjinKwon closed pull request #33139: [SPARK-35938][PYTHON] Add deprecation warning for Python 3.6

HyukjinKwon closed pull request #33139: URL: https://github.com/apache/spark/pull/33139 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-un

[GitHub] [spark] HyukjinKwon closed pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

HyukjinKwon closed pull request #33141: URL: https://github.com/apache/spark/pull/33141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-un

[GitHub] [spark] HyukjinKwon commented on pull request #33139: [SPARK-35938][PYTHON] Add deprecation warning for Python 3.6

HyukjinKwon commented on pull request #33139: URL: https://github.com/apache/spark/pull/33139#issuecomment-871816914 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [spark] HyukjinKwon commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

HyukjinKwon commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871816876 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [spark] SparkQA commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871815390 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44982/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

SparkQA commented on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-871812922 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44983/ -- This is an automated message from the Apache

[GitHub] [spark] HeartSaVioR commented on a change in pull request #33038: [SPARK-35861][SS] Introduce "prefix match scan" feature on state store

HeartSaVioR commented on a change in pull request #33038: URL: https://github.com/apache/spark/pull/33038#discussion_r661889937 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala ## @@ -252,6 +238,12 @@ priva

[GitHub] [spark] HeartSaVioR commented on a change in pull request #33038: [SPARK-35861][SS] Introduce "prefix match scan" feature on state store

HeartSaVioR commented on a change in pull request #33038: URL: https://github.com/apache/spark/pull/33038#discussion_r661889773 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala ## @@ -0,0 +1,172 @@ +/* + * Licen

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

dongjoon-hyun commented on a change in pull request #33160: URL: https://github.com/apache/spark/pull/33160#discussion_r661886492 ## File path: pom.xml ## @@ -3300,6 +3300,15 @@ + + no-shaded-client Review comment: +1 for the naming. -- Thi

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

dongjoon-hyun commented on a change in pull request #33160: URL: https://github.com/apache/spark/pull/33160#discussion_r661886492 ## File path: pom.xml ## @@ -3300,6 +3300,15 @@ + + no-shaded-client Review comment: +1 -- This is an automate

[GitHub] [spark] SparkQA commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

SparkQA commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871805138 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44981/ -- This is an automated message from the A

[GitHub] [spark] xinrong-databricks edited a comment on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

xinrong-databricks edited a comment on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871804786 FYI @HyukjinKwon @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [spark] xinrong-databricks commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

xinrong-databricks commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871804786 FYI @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

[GitHub] [spark] SparkQA commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871803991 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44980/ -- This is an automated message from the A

[GitHub] [spark] sunchao commented on a change in pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

sunchao commented on a change in pull request #33160: URL: https://github.com/apache/spark/pull/33160#discussion_r661884327 ## File path: pom.xml ## @@ -3300,6 +3300,15 @@ + + no-shaded-client Review comment: How about `no-shaded-hadoop-client`?

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

dongjoon-hyun commented on a change in pull request #33160: URL: https://github.com/apache/spark/pull/33160#discussion_r661883658 ## File path: pom.xml ## @@ -3300,6 +3300,15 @@ + + no-shaded-client Review comment: The name looks too general. Th

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

dongjoon-hyun commented on a change in pull request #33160: URL: https://github.com/apache/spark/pull/33160#discussion_r661883658 ## File path: pom.xml ## @@ -3300,6 +3300,15 @@ + + no-shaded-client Review comment: The name looks too general. Th

[GitHub] [spark] SparkQA commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871801361 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44978/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871798941 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44982/ -- This is an automated message from the Apache

[GitHub] [spark] zhouyejoe commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the sta

zhouyejoe commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661878682 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -403,38 +394,78 @@ public MergeS

[GitHub] [spark] otterc commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

otterc commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661875458 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -403,38 +394,78 @@ public MergeStat

[GitHub] [spark] zhouyejoe commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the sta

zhouyejoe commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661877750 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -403,38 +394,78 @@ public MergeS

[GitHub] [spark] otterc commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

otterc commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661875458 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -403,38 +394,78 @@ public MergeStat

[GitHub] [spark] SparkQA commented on pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

SparkQA commented on pull request #33160: URL: https://github.com/apache/spark/pull/33160#issuecomment-871794553 **[Test build #140469 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140469/testReport)** for PR 33160 at commit [`0bf197b`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32753: [SPARK-34859][SQL] Handle column index when using vectorized Parquet reader

AmplabJenkins removed a comment on pull request #32753: URL: https://github.com/apache/spark/pull/32753#issuecomment-871794031 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140459/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

AmplabJenkins removed a comment on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871777295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33158: [SPARK-35888][SQL][FOLLOWUP] Return partition specs for all the shuffles

AmplabJenkins removed a comment on pull request #33158: URL: https://github.com/apache/spark/pull/33158#issuecomment-871794036 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140457/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

AmplabJenkins removed a comment on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871794032 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44979/

[GitHub] [spark] AmplabJenkins commented on pull request #32753: [SPARK-34859][SQL] Handle column index when using vectorized Parquet reader

AmplabJenkins commented on pull request #32753: URL: https://github.com/apache/spark/pull/32753#issuecomment-871794031 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140459/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

AmplabJenkins commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871794037 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [spark] AmplabJenkins commented on pull request #33158: [SPARK-35888][SQL][FOLLOWUP] Return partition specs for all the shuffles

AmplabJenkins commented on pull request #33158: URL: https://github.com/apache/spark/pull/33158#issuecomment-871794036 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140457/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

AmplabJenkins commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871794032 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44979/ -- T

[GitHub] [spark] SparkQA commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

SparkQA commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871793472 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44981/ -- This is an automated message from the Apache

[GitHub] [spark] otterc commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

otterc commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661875458 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -403,38 +394,78 @@ public MergeStat

[GitHub] [spark] sunchao opened a new pull request #33160: [SPARK-35959][BUILD] Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions

sunchao opened a new pull request #33160: URL: https://github.com/apache/spark/pull/33160 ### What changes were proposed in this pull request? Add a new Maven profile `no-shaded-client` that, when activated, switches to non-shaded Hadoop client (e.g., `hadoop-client`, `ha

[GitHub] [spark] SparkQA commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871792172 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44980/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon commented on pull request #33146: [SPARK-35912][SQL] Fix cast struct contains null value to string/struct

HyukjinKwon commented on pull request #33146: URL: https://github.com/apache/spark/pull/33146#issuecomment-871792001 Hey mind explaining why cast path issue is related to being cached? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871789376 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44978/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

SparkQA commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871788655 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44979/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #32753: [SPARK-34859][SQL] Handle column index when using vectorized Parquet reader

SparkQA removed a comment on pull request #32753: URL: https://github.com/apache/spark/pull/32753#issuecomment-871646642 **[Test build #140459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140459/testReport)** for PR 32753 at commit [`6541d99`](https://gi

[GitHub] [spark] SparkQA commented on pull request #32753: [SPARK-34859][SQL] Handle column index when using vectorized Parquet reader

SparkQA commented on pull request #32753: URL: https://github.com/apache/spark/pull/32753#issuecomment-871784004 **[Test build #140459 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140459/testReport)** for PR 32753 at commit [`6541d99`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA removed a comment on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871777374 **[Test build #140468 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140468/testReport)** for PR 33141 at commit [`5123c6a`](https://gi

[GitHub] [spark] SparkQA commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871782552 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

[GitHub] [spark] SparkQA removed a comment on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA removed a comment on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871772010 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] SparkQA removed a comment on pull request #33158: [SPARK-35888][SQL][FOLLOWUP] Return partition specs for all the shuffles

SparkQA removed a comment on pull request #33158: URL: https://github.com/apache/spark/pull/33158#issuecomment-871639323 **[Test build #140457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140457/testReport)** for PR 33158 at commit [`f409ff6`](https://gi

[GitHub] [spark] SparkQA commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871779505 **[Test build #140466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140466/testReport)** for PR 33141 at commit [`e5c91d5`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #33158: [SPARK-35888][SQL][FOLLOWUP] Return partition specs for all the shuffles

SparkQA commented on pull request #33158: URL: https://github.com/apache/spark/pull/33158#issuecomment-871777586 **[Test build #140457 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140457/testReport)** for PR 33158 at commit [`f409ff6`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871777374 **[Test build #140468 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140468/testReport)** for PR 33141 at commit [`5123c6a`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

AmplabJenkins commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871777295 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140464/ -- This

[GitHub] [spark] SparkQA commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871777133 **[Test build #140464 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140464/testReport)** for PR 33141 at commit [`0bf064e`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #33139: [SPARK-35938][PYTHON] Add deprecation warning for Python 3.6

AmplabJenkins commented on pull request #33139: URL: https://github.com/apache/spark/pull/33139#issuecomment-871777048 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44976/ -- T

[GitHub] [spark] SparkQA commented on pull request #33139: [SPARK-35938][PYTHON] Add deprecation warning for Python 3.6

SparkQA commented on pull request #33139: URL: https://github.com/apache/spark/pull/33139#issuecomment-871777030 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44976/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

SparkQA commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871776039 **[Test build #140467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140467/testReport)** for PR 33093 at commit [`7629d2e`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #33141: [SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871774737 **[Test build #140466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140466/testReport)** for PR 33141 at commit [`e5c91d5`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #33141: [WIP][SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

SparkQA commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871772010 **[Test build #140464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140464/testReport)** for PR 33141 at commit [`0bf064e`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

SparkQA commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871772043 **[Test build #140465 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140465/testReport)** for PR 33093 at commit [`ef50c79`](https://github.com

[GitHub] [spark] otterc commented on a change in pull request #33034: WIP: [SPARK-32923][CORE][SHUFFLE] Handle indeterminate stage retries for push-based shuffle

otterc commented on a change in pull request #33034: URL: https://github.com/apache/spark/pull/33034#discussion_r661852708 ## File path: common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java ## @@ -222,7 +223,7 @@ public void sendMergedBlockM

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33141: [WIP][SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

AmplabJenkins removed a comment on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871770940 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140463/ -

[GitHub] [spark] AmplabJenkins commented on pull request #33141: [WIP][SPARK-35939][DOCS][PYTHON] Deprecate Python 3.6 in Spark documentation

AmplabJenkins commented on pull request #33141: URL: https://github.com/apache/spark/pull/33141#issuecomment-871770940 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140463/ -- This

[GitHub] [spark] otterc commented on a change in pull request #33034: WIP: [SPARK-32923][CORE][SHUFFLE] Handle indeterminate stage retries for push-based shuffle

otterc commented on a change in pull request #33034: URL: https://github.com/apache/spark/pull/33034#discussion_r661852708 ## File path: common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java ## @@ -222,7 +223,7 @@ public void sendMergedBlockM

<    1   2   3   4   5   6   7   8   >