[GitHub] [spark] SparkQA commented on pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

SparkQA commented on pull request #33410: URL: https://github.com/apache/spark/pull/33410#issuecomment-882261200 **[Test build #141226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141226/testReport)** for PR 33410 at commit

[GitHub] [spark] SparkQA commented on pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

SparkQA commented on pull request #33410: URL: https://github.com/apache/spark/pull/33410#issuecomment-882259548 **[Test build #141225 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141225/testReport)** for PR 33410 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

AmplabJenkins removed a comment on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882256940 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

SparkQA commented on pull request #33410: URL: https://github.com/apache/spark/pull/33410#issuecomment-882258068 **[Test build #141224 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141224/testReport)** for PR 33410 at commit

[GitHub] [spark] SparkQA commented on pull request #33411: [SPARK-36166][TESTS][FOLLOW-UP] Add Scala version change logic into testing script

SparkQA commented on pull request #33411: URL: https://github.com/apache/spark/pull/33411#issuecomment-882258032 **[Test build #141223 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141223/testReport)** for PR 33411 at commit

[GitHub] [spark] SparkQA commented on pull request #33412: [SPARK-36205][INFRA] Use set-env instead of set-output in GitHub Actions

SparkQA commented on pull request #33412: URL: https://github.com/apache/spark/pull/33412#issuecomment-882257973 **[Test build #141222 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141222/testReport)** for PR 33412 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

AmplabJenkins commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882256941 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

HyukjinKwon commented on a change in pull request #33410: URL: https://github.com/apache/spark/pull/33410#discussion_r671997638 ## File path: .github/workflows/build_and_test.yml ## @@ -5,11 +5,46 @@ on: branches: - '**' - '!branch-*.*' + schedule: +#

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

HyukjinKwon commented on a change in pull request #33410: URL: https://github.com/apache/spark/pull/33410#discussion_r671996885 ## File path: .github/workflows/build_and_test.yml ## @@ -5,11 +5,46 @@ on: branches: - '**' - '!branch-*.*' + schedule: +#

[GitHub] [spark] SparkQA commented on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

SparkQA commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882252237 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45731/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

SparkQA commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882252144 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45733/ --

[GitHub] [spark] SparkQA commented on pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

SparkQA commented on pull request #33410: URL: https://github.com/apache/spark/pull/33410#issuecomment-882251360 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45732/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

HyukjinKwon commented on a change in pull request #33410: URL: https://github.com/apache/spark/pull/33410#discussion_r671997638 ## File path: .github/workflows/build_and_test.yml ## @@ -5,11 +5,46 @@ on: branches: - '**' - '!branch-*.*' + schedule: +#

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

HyukjinKwon commented on a change in pull request #33410: URL: https://github.com/apache/spark/pull/33410#discussion_r671996885 ## File path: .github/workflows/build_and_test.yml ## @@ -5,11 +5,46 @@ on: branches: - '**' - '!branch-*.*' + schedule: +#

[GitHub] [spark] dongjoon-hyun commented on pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

dongjoon-hyun commented on pull request #33410: URL: https://github.com/apache/spark/pull/33410#issuecomment-882249751 @HyukjinKwon . If this works, I will not fix the newly recognized issue: The changed module detection was between `HEAD` and `HEAD~1` at the previous Scala 2.13 daily

[GitHub] [spark] dongjoon-hyun closed pull request #33403: [SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop SparkContext in non-K8s env

dongjoon-hyun closed pull request #33403: URL: https://github.com/apache/spark/pull/33403 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #33403: [SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop SparkContext in non-K8s env

dongjoon-hyun edited a comment on pull request #33403: URL: https://github.com/apache/spark/pull/33403#issuecomment-882248958 Thank you, @Ngone51 . Merged to master/3.2/3.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] dongjoon-hyun commented on pull request #33403: [SPARK-36193][CORE] Recover SparkSubmit.runMain not to stop SparkContext in non-K8s env

dongjoon-hyun commented on pull request #33403: URL: https://github.com/apache/spark/pull/33403#issuecomment-882248958 Thank you, @Ngone51 . Merged to master/3.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

dongjoon-hyun commented on a change in pull request #33410: URL: https://github.com/apache/spark/pull/33410#discussion_r671995134 ## File path: .github/workflows/build_and_test.yml ## @@ -5,11 +5,46 @@ on: branches: - '**' - '!branch-*.*' + schedule: +#

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

dongjoon-hyun commented on a change in pull request #33410: URL: https://github.com/apache/spark/pull/33410#discussion_r671995134 ## File path: .github/workflows/build_and_test.yml ## @@ -5,11 +5,46 @@ on: branches: - '**' - '!branch-*.*' + schedule: +#

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

dongjoon-hyun commented on a change in pull request #33410: URL: https://github.com/apache/spark/pull/33410#discussion_r671994626 ## File path: .github/workflows/build_and_test.yml ## @@ -5,11 +5,46 @@ on: branches: - '**' - '!branch-*.*' + schedule: +#

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

dongjoon-hyun commented on a change in pull request #33410: URL: https://github.com/apache/spark/pull/33410#discussion_r671994379 ## File path: .github/workflows/build_and_test.yml ## @@ -5,11 +5,46 @@ on: branches: - '**' - '!branch-*.*' + schedule: +#

[GitHub] [spark] HyukjinKwon commented on pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

HyukjinKwon commented on pull request #33410: URL: https://github.com/apache/spark/pull/33410#issuecomment-882246427 (This is still WIP BTW. I am just trying an idea) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] HyukjinKwon commented on pull request #33412: [MINOR][INFRA] Use set-env instead of set-output in GitHub Actions

HyukjinKwon commented on pull request #33412: URL: https://github.com/apache/spark/pull/33412#issuecomment-882246067 Oh sure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon commented on pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

HyukjinKwon commented on pull request #33410: URL: https://github.com/apache/spark/pull/33410#issuecomment-882245925 cc @dongjoon-hyun FYI. I think I am giving a shot to deduplicate the job here. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dongjoon-hyun commented on pull request #33412: [MINOR][INFRA] Use set-env instead of set-output in GitHub Actions

dongjoon-hyun commented on pull request #33412: URL: https://github.com/apache/spark/pull/33412#issuecomment-882245792 BTW, it would be great if we have a JIRA ID because it looks worth of backporting. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] dongjoon-hyun commented on pull request #33408: [SPARK-36199][BUILD] Bump scalatest-maven-plugin to 2.0.2

dongjoon-hyun commented on pull request #33408: URL: https://github.com/apache/spark/pull/33408#issuecomment-882245045 Thank you, @williamhyun and @HyukjinKwon . Merged to master/3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun closed pull request #33408: [SPARK-36199][BUILD] Bump scalatest-maven-plugin to 2.0.2

dongjoon-hyun closed pull request #33408: URL: https://github.com/apache/spark/pull/33408 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on pull request #33358: [SPARK-36152][INFRA][TESTS] Add Scala 2.13 daily build and test GitHub Action job

HyukjinKwon commented on pull request #33358: URL: https://github.com/apache/spark/pull/33358#issuecomment-882244494 No problem nothing blocks dev and PRs  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] dongjoon-hyun commented on pull request #33405: [SPARK-36195][BUILD] Set MaxMetaspaceSize JVM option to 2g

dongjoon-hyun commented on pull request #33405: URL: https://github.com/apache/spark/pull/33405#issuecomment-882244343 Sorry guys. I'm taking my word back. I'm check the job again. :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #33405: [SPARK-36195][BUILD] Set MaxMetaspaceSize JVM option to 2g

dongjoon-hyun edited a comment on pull request #33405: URL: https://github.com/apache/spark/pull/33405#issuecomment-882242140 ~FYI, `Scala 2.13` daily test GitHub Action job passed with this. Now, Apache Spark community have a regular Scala 2.13 test coverage.~ -

[GitHub] [spark] dongjoon-hyun commented on pull request #33358: [SPARK-36152][INFRA][TESTS] Add Scala 2.13 daily build and test GitHub Action job

dongjoon-hyun commented on pull request #33358: URL: https://github.com/apache/spark/pull/33358#issuecomment-882243876 Oh, sorry. I'm taking it back. It checked the updated module with the very previous commit. So, the job actually didn't run the Hive module actually. Let me check and fix

[GitHub] [spark] HyukjinKwon commented on pull request #33358: [SPARK-36152][INFRA][TESTS] Add Scala 2.13 daily build and test GitHub Action job

HyukjinKwon commented on pull request #33358: URL: https://github.com/apache/spark/pull/33358#issuecomment-882243699 Yeah, thanks for setting it up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] HyukjinKwon commented on pull request #33405: [SPARK-36195][BUILD] Set MaxMetaspaceSize JVM option to 2g

HyukjinKwon commented on pull request #33405: URL: https://github.com/apache/spark/pull/33405#issuecomment-882243495 Awesome! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #33358: [SPARK-36152][INFRA][TESTS] Add Scala 2.13 daily build and test GitHub Action job

dongjoon-hyun edited a comment on pull request #33358: URL: https://github.com/apache/spark/pull/33358#issuecomment-882242593 ~FYI, Scala 2.13 daily test GitHub Action job passed like the following. Now, Apache Spark community have a regular Scala 2.13 test coverage.~

[GitHub] [spark] dongjoon-hyun commented on pull request #33358: [SPARK-36152][INFRA][TESTS] Add Scala 2.13 daily build and test GitHub Action job

dongjoon-hyun commented on pull request #33358: URL: https://github.com/apache/spark/pull/33358#issuecomment-882242593 FYI, Scala 2.13 daily test GitHub Action job passed like the following. Now, Apache Spark community have a regular Scala 2.13 test coverage.

[GitHub] [spark] dongjoon-hyun commented on pull request #33405: [SPARK-36195][BUILD] Set MaxMetaspaceSize JVM option to 2g

dongjoon-hyun commented on pull request #33405: URL: https://github.com/apache/spark/pull/33405#issuecomment-882242140 FYI, `Scala 2.13` daily test GitHub Action job passed with this. Now, Apache Spark community have a regular Scala 2.13 test coverage. -

[GitHub] [spark] HyukjinKwon opened a new pull request #33412: [MINOR][INFRA] Use set-env instead of set-output in GitHub Actions

HyukjinKwon opened a new pull request #33412: URL: https://github.com/apache/spark/pull/33412 ### What changes were proposed in this pull request? This PR is more a cleanup. It removes unused `sync-branch` id in some steps, and use `set-env` instead of `set-output` to set an env.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33411: [SPARK-36166][TESTS][FOLLOW-UP] Add Scala version change logic into testing script

AmplabJenkins removed a comment on pull request #33411: URL: https://github.com/apache/spark/pull/33411#issuecomment-882239797 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141221/

[GitHub] [spark] SparkQA removed a comment on pull request #33411: [SPARK-36166][TESTS][FOLLOW-UP] Add Scala version change logic into testing script

SparkQA removed a comment on pull request #33411: URL: https://github.com/apache/spark/pull/33411#issuecomment-882239216 **[Test build #141221 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141221/testReport)** for PR 33411 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33411: [SPARK-36166][TESTS][FOLLOW-UP] Add Scala version change logic into testing script

AmplabJenkins commented on pull request #33411: URL: https://github.com/apache/spark/pull/33411#issuecomment-882239797 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141221/ -- This

[GitHub] [spark] SparkQA commented on pull request #33411: [SPARK-36166][TESTS][FOLLOW-UP] Add Scala version change logic into testing script

SparkQA commented on pull request #33411: URL: https://github.com/apache/spark/pull/33411#issuecomment-882239775 **[Test build #141221 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141221/testReport)** for PR 33411 at commit

[GitHub] [spark] SparkQA commented on pull request #33411: [SPARK-36166][TESTS][FOLLOW-UP] Add Scala version change logic into testing script

SparkQA commented on pull request #33411: URL: https://github.com/apache/spark/pull/33411#issuecomment-882239216 **[Test build #141221 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141221/testReport)** for PR 33411 at commit

[GitHub] [spark] HyukjinKwon opened a new pull request #33411: [SPARK-36166][TESTS][FOLLOW-UP] Add Scala version change logic into testing script

HyukjinKwon opened a new pull request #33411: URL: https://github.com/apache/spark/pull/33411 ### What changes were proposed in this pull request? This PR is a simple followup from https://github.com/apache/spark/pull/33376: - It simplifies a bit by removing the default Scala

[GitHub] [spark] zero323 commented on a change in pull request #33399: `udf` should return `UserDefinedFunction`

zero323 commented on a change in pull request #33399: URL: https://github.com/apache/spark/pull/33399#discussion_r671987962 ## File path: python/pyspark/sql/functions.pyi ## @@ -359,13 +360,13 @@ def variance(col: ColumnOrName) -> Column: ... @overload def udf( f:

[GitHub] [spark] SparkQA commented on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

SparkQA commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882237421 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45731/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33406: [SPARK-36197][SQL] Use PartitionDesc instead of TableDesc for reading hive partitioned tables

AmplabJenkins removed a comment on pull request #33406: URL: https://github.com/apache/spark/pull/33406#issuecomment-882233694 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33341: [SPARK-36091][SQL] Support TimestampNTZ type in expression TimeWindow

AmplabJenkins removed a comment on pull request #33341: URL: https://github.com/apache/spark/pull/33341#issuecomment-882233697 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45730/

[GitHub] [spark] SparkQA removed a comment on pull request #33406: [SPARK-36197][SQL] Use PartitionDesc instead of TableDesc for reading hive partitioned tables

SparkQA removed a comment on pull request #33406: URL: https://github.com/apache/spark/pull/33406#issuecomment-882191276 **[Test build #141216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141216/testReport)** for PR 33406 at commit

[GitHub] [spark] SparkQA commented on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

SparkQA commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882235105 **[Test build #141220 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141220/testReport)** for PR 33409 at commit

[GitHub] [spark] SparkQA commented on pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

SparkQA commented on pull request #33410: URL: https://github.com/apache/spark/pull/33410#issuecomment-882235068 **[Test build #141219 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141219/testReport)** for PR 33410 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33341: [SPARK-36091][SQL] Support TimestampNTZ type in expression TimeWindow

AmplabJenkins commented on pull request #33341: URL: https://github.com/apache/spark/pull/33341#issuecomment-882233697 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45730/ --

[GitHub] [spark] AmplabJenkins commented on pull request #33406: [SPARK-36197][SQL] Use PartitionDesc instead of TableDesc for reading hive partitioned tables

AmplabJenkins commented on pull request #33406: URL: https://github.com/apache/spark/pull/33406#issuecomment-882233694 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] SparkQA commented on pull request #33406: [SPARK-36197][SQL] Use PartitionDesc instead of TableDesc for reading hive partitioned tables

SparkQA commented on pull request #33406: URL: https://github.com/apache/spark/pull/33406#issuecomment-882231406 **[Test build #141216 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141216/testReport)** for PR 33406 at commit

[GitHub] [spark] HyukjinKwon opened a new pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

HyukjinKwon opened a new pull request #33410: URL: https://github.com/apache/spark/pull/33410 ### What changes were proposed in this pull request? Scala 2.13 daily job was added but ideally we should deduplicate it. This PR targets to deduplicate it by setting: - `0 0 * * *` for

[GitHub] [spark] SparkQA commented on pull request #33341: [SPARK-36091][SQL] Support TimestampNTZ type in expression TimeWindow

SparkQA commented on pull request #33341: URL: https://github.com/apache/spark/pull/33341#issuecomment-882223359 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45730/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #33406: [SPARK-36197][SQL] Use PartitionDesc instead of TableDesc for reading hive partitioned tables

SparkQA commented on pull request #33406: URL: https://github.com/apache/spark/pull/33406#issuecomment-88093 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45729/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

AmplabJenkins removed a comment on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882217431 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141218/

[GitHub] [spark] SparkQA removed a comment on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

SparkQA removed a comment on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882216558 **[Test build #141218 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141218/testReport)** for PR 33409 at commit

[GitHub] [spark] SparkQA commented on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

SparkQA commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882217416 **[Test build #141218 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141218/testReport)** for PR 33409 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

AmplabJenkins commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882217431 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141218/ -- This

[GitHub] [spark] SparkQA commented on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

SparkQA commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882216558 **[Test build #141218 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141218/testReport)** for PR 33409 at commit

[GitHub] [spark] AngersZhuuuu commented on pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

AngersZh commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-882216303 ping @dongjoon-hyun @HyukjinKwon @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] AngersZhuuuu opened a new pull request #33409: [SPARK-36021][SQL] Schema check should check inner field too

AngersZh opened a new pull request #33409: URL: https://github.com/apache/spark/pull/33409 ### What changes were proposed in this pull request? When inner field have wrong schema filed name should check field name too.

[GitHub] [spark] ulysses-you commented on a change in pull request #33396: [SPARK-36184][SQL] Use ValidateRequirements instead of EnsureRequirements to skip AQE rules that adds extra shuffles

ulysses-you commented on a change in pull request #33396: URL: https://github.com/apache/spark/pull/33396#discussion_r671972152 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CustomShuffleReaderRule.scala ## @@ -30,4 +30,6 @@ trait

[GitHub] [spark] itholic commented on pull request #33379: [SPARK-35810][PYTHON] Deprecate ps.broadcast API

itholic commented on pull request #33379: URL: https://github.com/apache/spark/pull/33379#issuecomment-882212025 > @itholic can you see if there are some pandas-on-Spark deprecations where we need to address the comment? Got it! -- This is an automated message from the Apache Git

[GitHub] [spark] Ngone51 commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

Ngone51 commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r671968278 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -112,34 +118,59 @@ public

[GitHub] [spark] SparkQA commented on pull request #33341: [SPARK-36091][SQL] Support TimestampNTZ type in expression TimeWindow

SparkQA commented on pull request #33341: URL: https://github.com/apache/spark/pull/33341#issuecomment-882209959 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45730/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33406: [SPARK-36197][SQL] Use PartitionDesc instead of TableDesc for reading hive partitioned tables

SparkQA commented on pull request #33406: URL: https://github.com/apache/spark/pull/33406#issuecomment-882209615 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45729/ -- This is an automated message from the Apache

[GitHub] [spark] Ngone51 commented on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

Ngone51 commented on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-882207137 Thanks @HeartSaVioR That's actually my concern. This PR only writes checksum but without verification. Verification is planned to be implemented in a separate PR and I worry we

[GitHub] [spark] HeartSaVioR commented on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

HeartSaVioR commented on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-882199840 I haven't looked into the code deeply (I just helped to fix RAT issue) but IMHO the value of this PR (worth to ship 3.2 or not) depends on whether we "verify" the checksum

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33408: [SPARK-36199][BUILD] Bump scalatest-maven-plugin to 2.0.2

AmplabJenkins removed a comment on pull request #33408: URL: https://github.com/apache/spark/pull/33408#issuecomment-882190916 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45728/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33407: [SPARK-36198][TESTS] Skip UNIDOC generation in PySpark GHA job

AmplabJenkins removed a comment on pull request #33407: URL: https://github.com/apache/spark/pull/33407#issuecomment-882190915 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141214/

[GitHub] [spark] SparkQA removed a comment on pull request #33407: [SPARK-36198][TESTS] Skip UNIDOC generation in PySpark GHA job

SparkQA removed a comment on pull request #33407: URL: https://github.com/apache/spark/pull/33407#issuecomment-882136741 **[Test build #141214 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141214/testReport)** for PR 33407 at commit

[GitHub] [spark] SparkQA commented on pull request #33341: [SPARK-36091][SQL] Support TimestampNTZ type in expression TimeWindow

SparkQA commented on pull request #33341: URL: https://github.com/apache/spark/pull/33341#issuecomment-882191281 **[Test build #141217 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141217/testReport)** for PR 33341 at commit

[GitHub] [spark] SparkQA commented on pull request #33406: [SPARK-36197][SQL] Use PartitionDesc instead of TableDesc for reading hive partitioned tables

SparkQA commented on pull request #33406: URL: https://github.com/apache/spark/pull/33406#issuecomment-882191276 **[Test build #141216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141216/testReport)** for PR 33406 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33407: [SPARK-36198][TESTS] Skip UNIDOC generation in PySpark GHA job

AmplabJenkins commented on pull request #33407: URL: https://github.com/apache/spark/pull/33407#issuecomment-882190915 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141214/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33408: [SPARK-36199][BUILD] Bump scalatest-maven-plugin to 2.0.2

AmplabJenkins commented on pull request #33408: URL: https://github.com/apache/spark/pull/33408#issuecomment-882190916 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45728/ --

[GitHub] [spark] Ngone51 commented on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

Ngone51 commented on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-882189754 @mridulm Thanks for the merge! I saw you also merged this into 3.2 but you may see that new confs in this PR are versioned as 3.3.0. Because my original thought was only

[GitHub] [spark] beliefer commented on pull request #33360: [SPARK-36090][SQL] Support TimestampNTZType in expression Sequence

beliefer commented on pull request #33360: URL: https://github.com/apache/spark/pull/33360#issuecomment-882187300 @MaxGekk Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA commented on pull request #33407: [SPARK-36198][TESTS] Skip UNIDOC generation in PySpark GHA job

SparkQA commented on pull request #33407: URL: https://github.com/apache/spark/pull/33407#issuecomment-882185712 **[Test build #141214 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141214/testReport)** for PR 33407 at commit

[GitHub] [spark] yaooqinn commented on pull request #33406: [SPARK-36197][SQL] Use PartitionDesc instead of TableDesc for reading hive partitioned tables

yaooqinn commented on pull request #33406: URL: https://github.com/apache/spark/pull/33406#issuecomment-882184074 cc @cloud-fan @maropu thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33363: [SPARK-36156][SQL] SCRIPT TRANSFORM ROW FORMAT DELIMITED should respect `NULL DEFINED AS` and default value should be `\N`

AngersZh commented on a change in pull request #33363: URL: https://github.com/apache/spark/pull/33363#discussion_r671944145 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala ## @@ -274,11 +278,21 @@ abstract class

[GitHub] [spark] AngersZhuuuu commented on pull request #33324: [SPARK-36093][SQL] RemoveRedundantAliases should not change Command's parameter's expression's name

AngersZh commented on pull request #33324: URL: https://github.com/apache/spark/pull/33324#issuecomment-882178752 Any more concern? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA commented on pull request #33408: [SPARK-36199][BUILD] Bump scalatest-maven-plugin to 2.0.2

SparkQA commented on pull request #33408: URL: https://github.com/apache/spark/pull/33408#issuecomment-882177842 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45728/ -- This is an automated message from the

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-882176352 Any more suggestion? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33407: [SPARK-36198][TESTS] Skip UNIDOC generation in PySpark GHA job

AmplabJenkins removed a comment on pull request #33407: URL: https://github.com/apache/spark/pull/33407#issuecomment-882167288 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45727/

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33399: `udf` should return `UserDefinedFunction`

HyukjinKwon commented on a change in pull request #33399: URL: https://github.com/apache/spark/pull/33399#discussion_r671938267 ## File path: python/pyspark/sql/functions.pyi ## @@ -359,13 +360,13 @@ def variance(col: ColumnOrName) -> Column: ... @overload def udf( f:

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33399: `udf` should return `UserDefinedFunction`

HyukjinKwon commented on a change in pull request #33399: URL: https://github.com/apache/spark/pull/33399#discussion_r671938214 ## File path: python/pyspark/sql/functions.pyi ## @@ -359,13 +360,13 @@ def variance(col: ColumnOrName) -> Column: ... @overload def udf( f:

[GitHub] [spark] AmplabJenkins commented on pull request #33407: [SPARK-36198][TESTS] Skip UNIDOC generation in PySpark GHA job

AmplabJenkins commented on pull request #33407: URL: https://github.com/apache/spark/pull/33407#issuecomment-882167288 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45727/ --

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33399: `udf` should return `UserDefinedFunction`

HyukjinKwon commented on a change in pull request #33399: URL: https://github.com/apache/spark/pull/33399#discussion_r671938014 ## File path: python/pyspark/sql/functions.pyi ## @@ -359,13 +360,13 @@ def variance(col: ColumnOrName) -> Column: ... @overload def udf( f:

[GitHub] [spark] HyukjinKwon closed pull request #33379: [SPARK-35810][PYTHON] Deprecate ps.broadcast API

HyukjinKwon closed pull request #33379: URL: https://github.com/apache/spark/pull/33379 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon edited a comment on pull request #33379: [SPARK-35810][PYTHON] Deprecate ps.broadcast API

HyukjinKwon edited a comment on pull request #33379: URL: https://github.com/apache/spark/pull/33379#issuecomment-882165800 Merged to master and branch-3.2. @itholic can you see if there are some pandas-on-Spark deprecations where we need to address the comment? -- This is an

[GitHub] [spark] HyukjinKwon commented on pull request #33379: [SPARK-35810][PYTHON] Deprecate ps.broadcast API

HyukjinKwon commented on pull request #33379: URL: https://github.com/apache/spark/pull/33379#issuecomment-882165800 Merged to master and branch-3.2. @itholic can you see if there are some deprecations where we need to address the comment? -- This is an automated message from the

[GitHub] [spark] HyukjinKwon commented on pull request #33407: [SPARK-36198][TESTS] Skip UNIDOC generation in PySpark GHA job

HyukjinKwon commented on pull request #33407: URL: https://github.com/apache/spark/pull/33407#issuecomment-882164829 I actually think you can also conditionally skip unidoc build (see `should_run_java_style_checks` in the script). It would be great if we can follow the existing way. We're

[GitHub] [spark] HyukjinKwon edited a comment on pull request #33407: [SPARK-36198][TESTS] Skip UNIDOC generation in PySpark GHA job

HyukjinKwon edited a comment on pull request #33407: URL: https://github.com/apache/spark/pull/33407#issuecomment-882162174 Yeah, nice. we should even skip the mima check. Would you be interested in working on that @williamhyun? -- This is an automated message from the Apache Git

[GitHub] [spark] HyukjinKwon commented on pull request #33407: [SPARK-36198][TESTS] Skip UNIDOC generation in PySpark GHA job

HyukjinKwon commented on pull request #33407: URL: https://github.com/apache/spark/pull/33407#issuecomment-882162174 Yeah, nice. we should even skip the scala/java style check. Would you be interested in working on that @williamhyun? -- This is an automated message from the Apache Git

[GitHub] [spark] SparkQA commented on pull request #33408: [SPARK-36199][BUILD] Bump scalatest-maven-plugin to 2.0.2

SparkQA commented on pull request #33408: URL: https://github.com/apache/spark/pull/33408#issuecomment-882161752 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45728/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon commented on pull request #33386: [SPARK-36174][SQL] Support explain final plan in AQE

HyukjinKwon commented on pull request #33386: URL: https://github.com/apache/spark/pull/33386#issuecomment-882161084 Yeah, I would avoid adding this for now either. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] ulysses-you commented on pull request #33211: [SPARK-36014][K8S] Use uuid as app id in kubernetes client mode

ulysses-you commented on pull request #33211: URL: https://github.com/apache/spark/pull/33211#issuecomment-882159225 thank you @dongjoon-hyun @holdenk ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

  1   2   3   >