[GitHub] [spark] AngersZhuuuu commented on pull request #32943: [SPARK-35735][SQL] Take into account day-time interval fields in cast

AngersZh commented on pull request #32943: URL: https://github.com/apache/spark/pull/32943#issuecomment-871052482 > @AngersZh You can re-trigger yourself by making an empty commit: > > ``` > git commit --allow-empty -m "Trigger build" > ``` Can work now, any more

[GitHub] [spark] AngersZhuuuu commented on pull request #32365: [SPARK-35228][SQL] Add expression ToHiveString for keep consistent between hive/spark format in df.show

AngersZh commented on pull request #32365: URL: https://github.com/apache/spark/pull/32365#issuecomment-871052319 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33144: [SPARK-35943][PYTHON] Introduce Axis type alias.

AmplabJenkins removed a comment on pull request #33144: URL: https://github.com/apache/spark/pull/33144#issuecomment-871052208 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44925/

[GitHub] [spark] AmplabJenkins commented on pull request #33145: Revert "[SPARK-34549][BUILD] Upgrade aws kinesis to 1.14.0 and java sdk 1.11.844"

AmplabJenkins commented on pull request #33145: URL: https://github.com/apache/spark/pull/33145#issuecomment-871052307 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44927/ -- T

[GitHub] [spark] SparkQA commented on pull request #33144: [SPARK-35943][PYTHON] Introduce Axis type alias.

SparkQA commented on pull request #33144: URL: https://github.com/apache/spark/pull/33144#issuecomment-871052197 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44925/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33144: [SPARK-35943][PYTHON] Introduce Axis type alias.

AmplabJenkins commented on pull request #33144: URL: https://github.com/apache/spark/pull/33144#issuecomment-871052208 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44925/ -- T

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33140: [SPARK-35881][SQL] Add support for columnar execution of final query stage in AdaptiveSparkPlanExec

AmplabJenkins removed a comment on pull request #33140: URL: https://github.com/apache/spark/pull/33140#issuecomment-870828466 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [spark] SparkQA commented on pull request #33028: [SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit

SparkQA commented on pull request #33028: URL: https://github.com/apache/spark/pull/33028#issuecomment-871051176 **[Test build #140418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140418/testReport)** for PR 33028 at commit [`54ed99e`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE

SparkQA commented on pull request #32850: URL: https://github.com/apache/spark/pull/32850#issuecomment-871051277 **[Test build #140419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140419/testReport)** for PR 32850 at commit [`d73bb83`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #33091: [SPARK-35896][SS] Include more granular metrics for stateful operators in StreamingQueryProgress

SparkQA commented on pull request #33091: URL: https://github.com/apache/spark/pull/33091#issuecomment-871051119 **[Test build #140417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140417/testReport)** for PR 33091 at commit [`c3f8a0d`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #33140: [SPARK-35881][SQL] Add support for columnar execution of final query stage in AdaptiveSparkPlanExec

SparkQA commented on pull request #33140: URL: https://github.com/apache/spark/pull/33140#issuecomment-871051085 **[Test build #140416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140416/testReport)** for PR 33140 at commit [`6ae47f4`](https://github.com

[GitHub] [spark] Ngone51 commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

Ngone51 commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661086353 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/PushBlockStream.java ## @@ -99,10 +110,11 @@ public void enc

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28032: [SPARK-31264][SQL] Repartition before writing data source tables/directories

AmplabJenkins removed a comment on pull request #28032: URL: https://github.com/apache/spark/pull/28032#issuecomment-871050550 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44917/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33095: [SPARK-35339][PYTHON] Improve unit tests for data-type-based basic operations

AmplabJenkins removed a comment on pull request #33095: URL: https://github.com/apache/spark/pull/33095#issuecomment-871050613 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140409/ -

[GitHub] [spark] AmplabJenkins commented on pull request #33095: [SPARK-35339][PYTHON] Improve unit tests for data-type-based basic operations

AmplabJenkins commented on pull request #33095: URL: https://github.com/apache/spark/pull/33095#issuecomment-871050613 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140409/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #28032: [SPARK-31264][SQL] Repartition before writing data source tables/directories

AmplabJenkins commented on pull request #28032: URL: https://github.com/apache/spark/pull/28032#issuecomment-871050550 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44917/ -- T

[GitHub] [spark] SparkQA removed a comment on pull request #33095: [SPARK-35339][PYTHON] Improve unit tests for data-type-based basic operations

SparkQA removed a comment on pull request #33095: URL: https://github.com/apache/spark/pull/33095#issuecomment-870967247 **[Test build #140409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140409/testReport)** for PR 33095 at commit [`1b48aec`](https://gi

[GitHub] [spark] karenfeng commented on a change in pull request #32850: [SPARK-34920][CORE][SQL] Add error classes with SQLSTATE

karenfeng commented on a change in pull request #32850: URL: https://github.com/apache/spark/pull/32850#discussion_r660908892 ## File path: core/src/main/scala/org/apache/spark/SparkError.scala ## @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

[GitHub] [spark] SparkQA commented on pull request #33095: [SPARK-35339][PYTHON] Improve unit tests for data-type-based basic operations

SparkQA commented on pull request #33095: URL: https://github.com/apache/spark/pull/33095#issuecomment-871049754 **[Test build #140409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140409/testReport)** for PR 33095 at commit [`1b48aec`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #32178: initial commit for skeleton ansible for jenkins worker config

SparkQA commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-871049299 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44919/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #32178: initial commit for skeleton ansible for jenkins worker config

SparkQA commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-871049220 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44923/ -- This is an automated message from the Apache

[GitHub] [spark] Ngone51 commented on pull request #33028: [SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit

Ngone51 commented on pull request #33028: URL: https://github.com/apache/spark/pull/33028#issuecomment-871048413 @dongjoon-hyun @HyukjinKwon @jiangxb1987 @mridulm anymore comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [spark] Ngone51 commented on a change in pull request #33028: [SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit

Ngone51 commented on a change in pull request #33028: URL: https://github.com/apache/spark/pull/33028#discussion_r661083366 ## File path: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala ## @@ -270,18 +270,20 @@ private[spark] class CoarseGraine

[GitHub] [spark] gaoyajun02 closed pull request #33131: [SPARK-35920][FOLLOWUP][BUILD] Fix Kryo Shaded dependency

gaoyajun02 closed pull request #33131: URL: https://github.com/apache/spark/pull/33131 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-uns

[GitHub] [spark] gaoyajun02 edited a comment on pull request #33131: [SPARK-35920][FOLLOWUP][BUILD] Fix Kryo Shaded dependency

gaoyajun02 edited a comment on pull request #33131: URL: https://github.com/apache/spark/pull/33131#issuecomment-871047161 I'm very sorry, when I executed the unit test today, there were no errors. I will close this PR. I have located the problem, because I am using my company's mave

[GitHub] [spark] SparkQA commented on pull request #32178: initial commit for skeleton ansible for jenkins worker config

SparkQA commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-871047578 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44921/ -- This is an automated message from the Apache

[GitHub] [spark] gaoyajun02 commented on pull request #33131: [SPARK-35920][FOLLOWUP][BUILD] Fix Kryo Shaded dependency

gaoyajun02 commented on pull request #33131: URL: https://github.com/apache/spark/pull/33131#issuecomment-871047161 I'm very sorry, when I executed the unit test today, there were no errors. I will close this PR. I have located the problem, because I am using my company's mvn mirror

[GitHub] [spark] SparkQA commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

SparkQA commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871046181 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44922/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

SparkQA commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871045704 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44920/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33012: [SPARK-33298][CORE] Introduce new API to FileCommitProtocol allow flexible file naming

HyukjinKwon commented on a change in pull request #33012: URL: https://github.com/apache/spark/pull/33012#discussion_r661081179 ## File path: core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala ## @@ -92,6 +92,35 @@ abstract class FileCommitProtocol exten

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33012: [SPARK-33298][CORE] Introduce new API to FileCommitProtocol allow flexible file naming

HyukjinKwon commented on a change in pull request #33012: URL: https://github.com/apache/spark/pull/33012#discussion_r661081115 ## File path: core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala ## @@ -92,6 +92,35 @@ abstract class FileCommitProtocol exten

[GitHub] [spark] xuanyuanking commented on pull request #32928: [SPARK-35784][SS] Implementation for RocksDB instance

xuanyuanking commented on pull request #32928: URL: https://github.com/apache/spark/pull/32928#issuecomment-871043411 Thanks all for the review! I'll rebase the other PRs soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] otterc closed pull request #33118: [WIP][SPARK-35917][SHUFFLE][CORE]Disable push-based shuffle feature to prevent it from being used

otterc closed pull request #33118: URL: https://github.com/apache/spark/pull/33118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubsc

[GitHub] [spark] otterc commented on pull request #33118: [WIP][SPARK-35917][SHUFFLE][CORE]Disable push-based shuffle feature to prevent it from being used

otterc commented on pull request #33118: URL: https://github.com/apache/spark/pull/33118#issuecomment-871042434 Makes sense @dongjoon-hyun @mridulm @HyukjinKwon. Will close this PR and create it against 3.2 once it is cut. -- This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA commented on pull request #28032: [SPARK-31264][SQL] Repartition before writing data source tables/directories

SparkQA commented on pull request #28032: URL: https://github.com/apache/spark/pull/28032#issuecomment-871041107 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44917/ -- This is an automated message from the A

[GitHub] [spark] dongjoon-hyun commented on pull request #33140: [SPARK-35881][SQL] Add support for columnar execution of final query stage in AdaptiveSparkPlanExec

dongjoon-hyun commented on pull request #33140: URL: https://github.com/apache/spark/pull/33140#issuecomment-871041118 Thank you, @andygrove . cc @sunchao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [spark] dongjoon-hyun commented on pull request #33140: [SPARK-35881][SQL] Add support for columnar execution of final query stage in AdaptiveSparkPlanExec

dongjoon-hyun commented on pull request #33140: URL: https://github.com/apache/spark/pull/33140#issuecomment-871040917 ok to test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[GitHub] [spark] dongjoon-hyun commented on pull request #33145: Revert "[SPARK-34549][BUILD] Upgrade aws kinesis to 1.14.0 and java sdk 1.11.844"

dongjoon-hyun commented on pull request #33145: URL: https://github.com/apache/spark/pull/33145#issuecomment-871039680 Thank you, @sarutak and @HyukjinKwon ! cc @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] Ngone51 commented on a change in pull request #33028: [SPARK-35714][FOLLOW-UP][CORE] Use a shared stopping flag for WorkerWatcher to avoid the duplicate System.exit

Ngone51 commented on a change in pull request #33028: URL: https://github.com/apache/spark/pull/33028#discussion_r661074976 ## File path: core/src/main/scala/org/apache/spark/deploy/worker/WorkerWatcher.scala ## @@ -53,10 +53,9 @@ private[spark] class WorkerWatcher( private

[GitHub] [spark] HyukjinKwon commented on pull request #33118: [WIP][SPARK-35917][SHUFFLE][CORE]Disable push-based shuffle feature to prevent it from being used

HyukjinKwon commented on pull request #33118: URL: https://github.com/apache/spark/pull/33118#issuecomment-871038469 only few days left for the branchcut :-). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [spark] HyukjinKwon commented on pull request #33118: [WIP][SPARK-35917][SHUFFLE][CORE]Disable push-based shuffle feature to prevent it from being used

HyukjinKwon commented on pull request #33118: URL: https://github.com/apache/spark/pull/33118#issuecomment-871038338 Yeah, +1 for landing this to branch-3.2 only -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] HyukjinKwon commented on pull request #33118: [WIP][SPARK-35917][SHUFFLE][CORE]Disable push-based shuffle feature to prevent it from being used

HyukjinKwon commented on pull request #33118: URL: https://github.com/apache/spark/pull/33118#issuecomment-871037747 cc @Ngone51 FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] vkorukanti commented on a change in pull request #33091: [SPARK-35896][SS] Include more granular metrics for stateful operators in StreamingQueryProgress

vkorukanti commented on a change in pull request #33091: URL: https://github.com/apache/spark/pull/33091#discussion_r661073820 ## File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala ## @@ -1122,6 +1106,43 @@ class FlatMapGroupsWi

[GitHub] [spark] SparkQA commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

SparkQA commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871035782 **[Test build #140415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140415/testReport)** for PR 33093 at commit [`673008d`](https://github.com

[GitHub] [spark] mridulm commented on pull request #32902: [SPARK-35754][CORE] Add config to put migrating blocks on disk only

mridulm commented on pull request #32902: URL: https://github.com/apache/spark/pull/32902#issuecomment-871035308 This would be inconsistent with the expectation of the storage level set on the RDD/DF. Instead, what we can do is prefer disk in case it is available as an option (if `stora

[GitHub] [spark] HyukjinKwon closed pull request #33144: [SPARK-35943][PYTHON] Introduce Axis type alias.

HyukjinKwon closed pull request #33144: URL: https://github.com/apache/spark/pull/33144 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-un

[GitHub] [spark] HyukjinKwon commented on pull request #33144: [SPARK-35943][PYTHON] Introduce Axis type alias.

HyukjinKwon commented on pull request #33144: URL: https://github.com/apache/spark/pull/33144#issuecomment-871034877 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [spark] HyukjinKwon closed pull request #33145: Revert "[SPARK-34549][BUILD] Upgrade aws kinesis to 1.14.0 and java sdk 1.11.844"

HyukjinKwon closed pull request #33145: URL: https://github.com/apache/spark/pull/33145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-un

[GitHub] [spark] HyukjinKwon commented on pull request #33145: Revert "[SPARK-34549][BUILD] Upgrade aws kinesis to 1.14.0 and java sdk 1.11.844"

HyukjinKwon commented on pull request #33145: URL: https://github.com/apache/spark/pull/33145#issuecomment-871034413 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

[GitHub] [spark] mridulm edited a comment on pull request #33118: [WIP][SPARK-35917][SHUFFLE][CORE]Disable push-based shuffle feature to prevent it from being used

mridulm edited a comment on pull request #33118: URL: https://github.com/apache/spark/pull/33118#issuecomment-871034230 That makes sense, thanks for the excellent suggestion @dongjoon-hyun ! Let us close this PR @otterc and create it against branch-3.2 when it has been cut - we can merge

[GitHub] [spark] mridulm commented on pull request #33118: [WIP][SPARK-35917][SHUFFLE][CORE]Disable push-based shuffle feature to prevent it from being used

mridulm commented on pull request #33118: URL: https://github.com/apache/spark/pull/33118#issuecomment-871034230 That makes sense, let us close this PR @otterc and create it against branch-3.2 when it has been cut and if we determine that the two correctness fixes cant make it to 3.2 --

[GitHub] [spark] AmplabJenkins commented on pull request #33143: Revert "[SPARK-33995][SQL] Expose make_interval as a Scala function"

AmplabJenkins commented on pull request #33143: URL: https://github.com/apache/spark/pull/33143#issuecomment-871033291 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140402/ -- This

[GitHub] [spark] Yikun commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests

Yikun commented on pull request #32867: URL: https://github.com/apache/spark/pull/32867#issuecomment-871032645 Thanks for revert for it. I'm so sorry for the trouble, I will recheck and fix soon. -- This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [spark] SparkQA removed a comment on pull request #33143: Revert "[SPARK-33995][SQL] Expose make_interval as a Scala function"

SparkQA removed a comment on pull request #33143: URL: https://github.com/apache/spark/pull/33143#issuecomment-870898180 **[Test build #140402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140402/testReport)** for PR 33143 at commit [`33b5b9a`](https://gi

[GitHub] [spark] SparkQA commented on pull request #33143: Revert "[SPARK-33995][SQL] Expose make_interval as a Scala function"

SparkQA commented on pull request #33143: URL: https://github.com/apache/spark/pull/33143#issuecomment-871032334 **[Test build #140402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140402/testReport)** for PR 33143 at commit [`33b5b9a`](https://github.co

[GitHub] [spark] sarutak commented on pull request #33145: Revert "[SPARK-34549][BUILD] Upgrade aws kinesis to 1.14.0 and java sdk 1.11.844"

sarutak commented on pull request #33145: URL: https://github.com/apache/spark/pull/33145#issuecomment-871032042 Confirmed all the `*Kinesis*Suite` pass with the following command without newer Guava. ``` ENABLE_KINESIS_TESTS=1 KINESIS_TEST_ENDPOINT_URL=https://kinesis.ap-northeast-1

[GitHub] [spark] SparkQA commented on pull request #32883: [SPARK-35725][SQL] Support optimize skewed partitions in RebalancePartitions

SparkQA commented on pull request #32883: URL: https://github.com/apache/spark/pull/32883#issuecomment-871031005 **[Test build #140414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140414/testReport)** for PR 32883 at commit [`fd44620`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #33093: [SPARK-35897][SS][WIP] Support user defined initial state with flatMapGroupsWithState in Structured Streaming

SparkQA commented on pull request #33093: URL: https://github.com/apache/spark/pull/33093#issuecomment-871030899 **[Test build #140413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140413/testReport)** for PR 33093 at commit [`f221358`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32178: initial commit for skeleton ansible for jenkins worker config

AmplabJenkins removed a comment on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-871030778 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44918/

[GitHub] [spark] SparkQA commented on pull request #33145: Revert "[SPARK-34549][BUILD] Upgrade aws kinesis to 1.14.0 and java sdk 1.11.844"

SparkQA commented on pull request #33145: URL: https://github.com/apache/spark/pull/33145#issuecomment-871030849 **[Test build #140412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140412/testReport)** for PR 33145 at commit [`ef6f752`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #32178: initial commit for skeleton ansible for jenkins worker config

AmplabJenkins commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-871030778 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44918/ -- T

[GitHub] [spark] SparkQA commented on pull request #32178: initial commit for skeleton ansible for jenkins worker config

SparkQA commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-871030766 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44918/ -- This

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31790: [SPARK-34509][K8S] Make dynamic allocation upscaling more progressive on K8S

AmplabJenkins removed a comment on pull request #31790: URL: https://github.com/apache/spark/pull/31790#issuecomment-871030378 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44913/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache

AmplabJenkins removed a comment on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-871030376 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44914/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32980: [SPARK-35829][SQL] Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin

AmplabJenkins removed a comment on pull request #32980: URL: https://github.com/apache/spark/pull/32980#issuecomment-871014566 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33144: [SPARK-35943][PYTHON] Introduce Axis type alias.

AmplabJenkins removed a comment on pull request #33144: URL: https://github.com/apache/spark/pull/33144#issuecomment-871030381 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140410/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32298: [SPARK-34079][SQL] Merge non-correlated scalar subqueries for better reuse

AmplabJenkins removed a comment on pull request #32298: URL: https://github.com/apache/spark/pull/32298#issuecomment-871014688 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31490: [SPARK-34365][AVRO] Add support for positional Catalyst-to-Avro schema matching

AmplabJenkins removed a comment on pull request #31490: URL: https://github.com/apache/spark/pull/31490#issuecomment-871030375 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44915/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29326: [WIP][SPARK-32502][BUILD] Upgrade Guava to 27.0-jre

AmplabJenkins removed a comment on pull request #29326: URL: https://github.com/apache/spark/pull/29326#issuecomment-871030380 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44916/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32178: initial commit for skeleton ansible for jenkins worker config

AmplabJenkins removed a comment on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-871030372 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [spark] AmplabJenkins commented on pull request #32298: [SPARK-34079][SQL] Merge non-correlated scalar subqueries for better reuse

AmplabJenkins commented on pull request #32298: URL: https://github.com/apache/spark/pull/32298#issuecomment-871030379 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44909/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #31790: [SPARK-34509][K8S] Make dynamic allocation upscaling more progressive on K8S

AmplabJenkins commented on pull request #31790: URL: https://github.com/apache/spark/pull/31790#issuecomment-871030378 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44913/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #29326: [WIP][SPARK-32502][BUILD] Upgrade Guava to 27.0-jre

AmplabJenkins commented on pull request #29326: URL: https://github.com/apache/spark/pull/29326#issuecomment-871030380 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44916/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache

AmplabJenkins commented on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-871030376 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44914/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #31490: [SPARK-34365][AVRO] Add support for positional Catalyst-to-Avro schema matching

AmplabJenkins commented on pull request #31490: URL: https://github.com/apache/spark/pull/31490#issuecomment-871030375 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44915/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #32980: [SPARK-35829][SQL] Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin

AmplabJenkins commented on pull request #32980: URL: https://github.com/apache/spark/pull/32980#issuecomment-871030373 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140394/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33144: [SPARK-35943][PYTHON] Introduce Axis type alias.

AmplabJenkins commented on pull request #33144: URL: https://github.com/apache/spark/pull/33144#issuecomment-871030381 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140410/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #32178: initial commit for skeleton ansible for jenkins worker config

AmplabJenkins commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-871030377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [spark] mridulm commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

mridulm commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661066092 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -567,7 +598,8 @@ public void onDat

[GitHub] [spark] SparkQA commented on pull request #28032: [SPARK-31264][SQL] Repartition before writing data source tables/directories

SparkQA commented on pull request #28032: URL: https://github.com/apache/spark/pull/28032#issuecomment-871029818 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44917/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA removed a comment on pull request #32178: initial commit for skeleton ansible for jenkins worker config

SparkQA removed a comment on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-870963197 **[Test build #140408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140408/testReport)** for PR 32178 at commit [`a72946f`](https://gi

[GitHub] [spark] SparkQA commented on pull request #32178: initial commit for skeleton ansible for jenkins worker config

SparkQA commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-871028805 **[Test build #140408 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140408/testReport)** for PR 32178 at commit [`a72946f`](https://github.co

[GitHub] [spark] mridulm commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

mridulm commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661066092 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -567,7 +598,8 @@ public void onDat

[GitHub] [spark] HyukjinKwon commented on pull request #33143: Revert "[SPARK-33995][SQL] Expose make_interval as a Scala function"

HyukjinKwon commented on pull request #33143: URL: https://github.com/apache/spark/pull/33143#issuecomment-871028036 I think it's fine though since we didn't deprecate `CalendarIntervalType`. That specific type has been there from 1.5.0. -- This is an automated message from the Apache Gi

[GitHub] [spark] mridulm commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

mridulm commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661065178 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -403,38 +394,78 @@ public MergeSta

[GitHub] [spark] mridulm commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

mridulm commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661065178 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -403,38 +394,78 @@ public MergeSta

[GitHub] [spark] sarutak edited a comment on pull request #33145: Revert SPARK-34549

sarutak edited a comment on pull request #33145: URL: https://github.com/apache/spark/pull/33145#issuecomment-871025313 cc: @HyukjinKwon @dongjoon-hyun @viirya I'll check if all the kinesis tests work on AWS environment. -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] mridulm commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

mridulm commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661063124 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -403,38 +394,78 @@ public MergeSta

[GitHub] [spark] sarutak commented on pull request #33145: Revert SPARK-34549

sarutak commented on pull request #33145: URL: https://github.com/apache/spark/pull/33145#issuecomment-871025313 cc: @HyukjinKwon @dongjoon-hyun @viirya I'll check all the kinesis tests work on AWS environment. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] mridulm commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

mridulm commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661063124 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -403,38 +394,78 @@ public MergeSta

[GitHub] [spark] sarutak opened a new pull request #33145: Revert SPARK-34549

sarutak opened a new pull request #33145: URL: https://github.com/apache/spark/pull/33145 ### What changes were proposed in this pull request? This PR reverts the change of SPARK-34549 ( #31658). ### Why are the changes needed? See #33133. ### Does this PR introdu

[GitHub] [spark] mridulm commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

mridulm commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r661062558 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/PushBlockStream.java ## @@ -99,10 +110,11 @@ public void enc

[GitHub] [spark] ulysses-you commented on pull request #33123: [SPARK-35923][SQL] Coalesce empty partition with mixed CoalescedPartitionSpec and PartialReducerPartitionSpec

ulysses-you commented on pull request #33123: URL: https://github.com/apache/spark/pull/33123#issuecomment-871024309 thank you @cloud-fan @yaooqinn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] SparkQA removed a comment on pull request #33144: [SPARK-35943][PYTHON] Introduce Axis type alias.

SparkQA removed a comment on pull request #33144: URL: https://github.com/apache/spark/pull/33144#issuecomment-871010709 **[Test build #140410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140410/testReport)** for PR 33144 at commit [`63530c4`](https://gi

[GitHub] [spark] SparkQA commented on pull request #33144: [SPARK-35943][PYTHON] Introduce Axis type alias.

SparkQA commented on pull request #33144: URL: https://github.com/apache/spark/pull/33144#issuecomment-871023305 **[Test build #140410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140410/testReport)** for PR 33144 at commit [`63530c4`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #31490: [SPARK-34365][AVRO] Add support for positional Catalyst-to-Avro schema matching

SparkQA commented on pull request #31490: URL: https://github.com/apache/spark/pull/31490#issuecomment-871021915 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44915/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #32178: initial commit for skeleton ansible for jenkins worker config

SparkQA commented on pull request #32178: URL: https://github.com/apache/spark/pull/32178#issuecomment-871020281 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44911/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #31790: [SPARK-34509][K8S] Make dynamic allocation upscaling more progressive on K8S

SparkQA commented on pull request #31790: URL: https://github.com/apache/spark/pull/31790#issuecomment-871020245 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44913/ -- This is an automated message from the A

[GitHub] [spark] viirya commented on pull request #32980: [SPARK-35829][SQL] Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin

viirya commented on pull request #32980: URL: https://github.com/apache/spark/pull/32980#issuecomment-871020124 @maropu Any more comments? Otherwise I will merge this later. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [spark] HyukjinKwon commented on pull request #33128: [SPARK-35873][PYTHON] Cleanup the version logic from the pandas API on Spark

HyukjinKwon commented on pull request #33128: URL: https://github.com/apache/spark/pull/33128#issuecomment-871020002 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

<    1   2   3   4   5   6   7   8   9   10   >