[GitHub] [spark] AmplabJenkins removed a comment on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command

AmplabJenkins removed a comment on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775503100 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135044/ -

[GitHub] [spark] holdenk commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request

holdenk commented on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775509482 LGTM pending CI. Jenkins failure is unrelated but to be safe, Jenkins retest this please. This is an automate

[GitHub] [spark] SparkQA commented on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.

SparkQA commented on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-775509993 **[Test build #135048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135048/testReport)** for PR 31510 at commit [`bff7afd`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775510620 **[Test build #135047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135047/testReport)** for PR 31529 at commit [`1e3e524`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

SparkQA removed a comment on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775503416 **[Test build #135047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135047/testReport)** for PR 31529 at commit [`1e3e524`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks

SparkQA commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775514042 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39629/ ---

[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775517184 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39628/ ---

[GitHub] [spark] SparkQA commented on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.

SparkQA commented on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-775525435 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39631/ -

[GitHub] [spark] SparkQA commented on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command

SparkQA commented on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775526942 **[Test build #135043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135043/testReport)** for PR 31494 at commit [`8b5508b`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command

SparkQA removed a comment on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775390813 **[Test build #135043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135043/testReport)** for PR 31494 at commit [`8b5508b`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks

AmplabJenkins commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775528864 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39629/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command

AmplabJenkins commented on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775528860 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135043/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

AmplabJenkins commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775528859 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31494: [SPARK-34380][SQL] Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES for v2 command

AmplabJenkins removed a comment on pull request #31494: URL: https://github.com/apache/spark/pull/31494#issuecomment-775528860 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135043/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

AmplabJenkins removed a comment on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775528858 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks

AmplabJenkins removed a comment on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775528864 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39629/

[GitHub] [spark] SparkQA commented on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.

SparkQA commented on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-775536471 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39631/ ---

[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request

SparkQA commented on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775536734 **[Test build #135049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135049/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

SparkQA commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-775537236 **[Test build #135050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135050/testReport)** for PR 31249 at commit [`83c6e2e`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks

SparkQA commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775537415 **[Test build #135046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135046/testReport)** for PR 31493 at commit [`69b2540`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks

SparkQA removed a comment on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-775470550 **[Test build #135046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135046/testReport)** for PR 31493 at commit [`69b2540`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request

SparkQA commented on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775541886 **[Test build #135049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135049/testReport)** for PR 31445 at commit [`ef5d0e9`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request

SparkQA removed a comment on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775536734 **[Test build #135049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135049/testReport)** for PR 31445 at commit [`ef5d0e9`](https://gi

[GitHub] [spark] HyukjinKwon commented on pull request #31496: [SPARK-34384][CORE] API cleanup for ResourceProfile

HyukjinKwon commented on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-775543510 I don't think blocks RC I am preparing now. Same here too. Hope we can do this earlier next time. @tgravescs, though, some of them like https://github.com/apache/spar

[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request

SparkQA commented on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775552827 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39632/ -

[GitHub] [spark] MaxGekk commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

MaxGekk commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775553799 jenkins, retest this, please This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request

AmplabJenkins commented on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-77758 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135049/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks

AmplabJenkins commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-77757 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135046/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

AmplabJenkins commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-77761 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39630/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.

AmplabJenkins commented on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-77759 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39631/ -

[GitHub] [spark] SparkQA commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

SparkQA commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-775556423 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39633/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

AmplabJenkins removed a comment on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-77761 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39630/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.

AmplabJenkins removed a comment on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-77759 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39631/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request

AmplabJenkins removed a comment on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-77758 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135049/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31493: [SPARK-34363][CORE] Add an option for limiting storage for migrated shuffle blocks

AmplabJenkins removed a comment on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-77757 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135046/ -

[GitHub] [spark] SparkQA commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

SparkQA commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-775558803 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39633/ ---

[GitHub] [spark] AmplabJenkins commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

AmplabJenkins commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-775558824 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39633/ -

[GitHub] [spark] HeartSaVioR commented on a change in pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

HeartSaVioR commented on a change in pull request #31495: URL: https://github.com/apache/spark/pull/31495#discussion_r572473687 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala ## @@ -239,18 +239,35 @@ class HDFSMetadataLog[T

[GitHub] [spark] HeartSaVioR commented on a change in pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

HeartSaVioR commented on a change in pull request #31495: URL: https://github.com/apache/spark/pull/31495#discussion_r572473687 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala ## @@ -239,18 +239,35 @@ class HDFSMetadataLog[T

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

AmplabJenkins removed a comment on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-775558824 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39633/

[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775561922 **[Test build #135051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135051/testReport)** for PR 31529 at commit [`1e3e524`](https://github.com

[GitHub] [spark] HeartSaVioR commented on a change in pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

HeartSaVioR commented on a change in pull request #31495: URL: https://github.com/apache/spark/pull/31495#discussion_r572479885 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala ## @@ -239,18 +239,35 @@ class HDFSMetadataLog[T

[GitHub] [spark] HeartSaVioR commented on pull request #31520: [SPARK-34355][CORE][SQL][FOLLOWUP] Log commit time in all File Writer

HeartSaVioR commented on pull request #31520: URL: https://github.com/apache/spark/pull/31520#issuecomment-775566881 I'm sorry I haven't considered about the impact of extracting the code - logger. I'm not sure we are leveraging the information on classname difference for tracking log on c

[GitHub] [spark] github-actions[bot] closed pull request #30199: [SPARK-33296][SQL] Format exception of unsupported nested grouping analytics

github-actions[bot] closed pull request #30199: URL: https://github.com/apache/spark/pull/30199 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [spark] SparkQA commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request

SparkQA commented on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775568472 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39632/ ---

[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775571042 **[Test build #135051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135051/testReport)** for PR 31529 at commit [`1e3e524`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

SparkQA removed a comment on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775561922 **[Test build #135051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135051/testReport)** for PR 31529 at commit [`1e3e524`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775574877 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39634/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

AmplabJenkins commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775580868 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135051/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request

AmplabJenkins commented on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775580867 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39632/ -

[GitHub] [spark] SparkQA commented on pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

SparkQA commented on pull request #31495: URL: https://github.com/apache/spark/pull/31495#issuecomment-775582196 **[Test build #135052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135052/testReport)** for PR 31495 at commit [`03801f7`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

AmplabJenkins removed a comment on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775580868 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135051/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31445: [SPARK-34334][K8S] Correctly identify timed out pending pod requests as excess request

AmplabJenkins removed a comment on pull request #31445: URL: https://github.com/apache/spark/pull/31445#issuecomment-775580867 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39632/

[GitHub] [spark] SparkQA commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

SparkQA commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-775584209 **[Test build #135050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135050/testReport)** for PR 31249 at commit [`83c6e2e`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

SparkQA removed a comment on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-775537236 **[Test build #135050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135050/testReport)** for PR 31249 at commit [`83c6e2e`](https://gi

[GitHub] [spark] JohnHBauer opened a new pull request #31530: Added KMeansSummary to __all__ in clustering.py

JohnHBauer opened a new pull request #31530: URL: https://github.com/apache/spark/pull/31530 KMeansSummary was undocumented. The KMeansSummary class is present in the code, not exposed in __all__, and therefore not documented in the Sphinx build. Adding 'KMeansSummary' to __al

[GitHub] [spark] AmplabJenkins commented on pull request #31530: Added KMeansSummary to __all__ in clustering.py

AmplabJenkins commented on pull request #31530: URL: https://github.com/apache/spark/pull/31530#issuecomment-775584415 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

AmplabJenkins commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-775584697 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135050/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

AmplabJenkins removed a comment on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-775584697 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135050/ -

[GitHub] [spark] SparkQA commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

SparkQA commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775586246 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39634/ ---

[GitHub] [spark] viirya opened a new pull request #31531: [SPARK-34080][ML][PYTHON][FOLLOW-UP] Update score function in document

viirya opened a new pull request #31531: URL: https://github.com/apache/spark/pull/31531 ### What changes were proposed in this pull request? This follows up #31160 to update score function in the document. ### Why are the changes needed? Currently we use

[GitHub] [spark] viirya commented on pull request #31531: [SPARK-34080][ML][PYTHON][FOLLOW-UP] Update score function in document

viirya commented on pull request #31531: URL: https://github.com/apache/spark/pull/31531#issuecomment-775588020 cc @huaxingao @WeichenXu123 This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] Ngone51 commented on a change in pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

Ngone51 commented on a change in pull request #31495: URL: https://github.com/apache/spark/pull/31495#discussion_r572505384 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala ## @@ -239,18 +239,35 @@ class HDFSMetadataLog[T <:

[GitHub] [spark] Ngone51 commented on pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

Ngone51 commented on pull request #31495: URL: https://github.com/apache/spark/pull/31495#issuecomment-775589882 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] beliefer commented on pull request #31526: [SPARK-34157][BUILD][FOLLOW-UP] Fix Scala 2.13 compilation error via using Array.deep

beliefer commented on pull request #31526: URL: https://github.com/apache/spark/pull/31526#issuecomment-775595375 @HyukjinKwon Thanks for your fix! This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [spark] HyukjinKwon commented on pull request #31466: [SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run on windows system

HyukjinKwon commented on pull request #31466: URL: https://github.com/apache/spark/pull/31466#issuecomment-775597566 Merged to master and branch-3.1. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] HyukjinKwon closed pull request #31466: [SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run on windows system

HyukjinKwon closed pull request #31466: URL: https://github.com/apache/spark/pull/31466 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] beliefer commented on pull request #31519: [SPARK-34394][SQL] Unify output of SHOW FUNCTIONS and pass output attributes properly

beliefer commented on pull request #31519: URL: https://github.com/apache/spark/pull/31519#issuecomment-775597852 cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] beliefer commented on pull request #31508: [SPARK-34393][SQL] Unify output of SHOW VIEWS and pass output attributes properly

beliefer commented on pull request #31508: URL: https://github.com/apache/spark/pull/31508#issuecomment-775597953 cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] beliefer commented on pull request #31466: [SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run on windows system

beliefer commented on pull request #31466: URL: https://github.com/apache/spark/pull/31466#issuecomment-775599266 @HyukjinKwon Thanks for your work! @cloud-fan @maropu Thanks for your review. This is an automated message from

[GitHub] [spark] beliefer commented on a change in pull request #31466: [SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run on windows system

beliefer commented on a change in pull request #31466: URL: https://github.com/apache/spark/pull/31466#discussion_r572513892 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala ## @@ -154,10 +154,14 @@ class SQLQueryTestSuite extends QueryTest wit

[GitHub] [spark] mengxr commented on pull request #31496: [SPARK-34384][CORE] API cleanup for ResourceProfile

mengxr commented on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-775599608 @tgravescs I think there are two separate questions: 1. Does it block 3.1 release? I agree with you that it shouldn't. It is indeed too late in the release process and the

[GitHub] [spark] mengxr edited a comment on pull request #31496: [SPARK-34384][CORE] API cleanup for ResourceProfile

mengxr edited a comment on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-775599608 @tgravescs I think there are two separate questions: 1. Does it block 3.1 release? I agree with you that it shouldn't. It is indeed too late in the release process a

[GitHub] [spark] SparkQA commented on pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

SparkQA commented on pull request #31495: URL: https://github.com/apache/spark/pull/31495#issuecomment-775601900 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39635/ -

[GitHub] [spark] LuciferYang commented on pull request #31484: [SPARK-34374][SQL][DSTREAM] Use standard methods to extract keys or values from a Map

LuciferYang commented on pull request #31484: URL: https://github.com/apache/spark/pull/31484#issuecomment-775605089 thx @srowen @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats

AngersZh commented on a change in pull request #31485: URL: https://github.com/apache/spark/pull/31485#discussion_r572518676 ## File path: sql/core/src/test/resources/sql-tests/inputs/explain-cbo.sql ## @@ -0,0 +1,25 @@ +CREATE TABLE t1(a INT, b INT) USING PARQUET; +CREATE

[GitHub] [spark] AmplabJenkins commented on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

AmplabJenkins commented on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775605752 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39634/ -

[GitHub] [spark] AngersZhuuuu commented on pull request #31516: [SPARK-34238][SQL][FOLLOW_UP] SHOW PARTITIONS Keep consistence with other `SHOW` command

AngersZh commented on pull request #31516: URL: https://github.com/apache/spark/pull/31516#issuecomment-775606160 retest this please This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [spark] LuciferYang commented on pull request #31517: [WIP][SPARK-34309][CORE][SQL] Use Caffeine instead of Guava Cache

LuciferYang commented on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-775607120 @Ngone51 I have tested some benchmark data in [SPARK-34309](https://issues.apache.org/jira/browse/SPARK-34309?focusedCommentId=17279548&page=com.atlassian.jira.plugin.system.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31529: [SPARK-34404][SQL] Add new Avro datasource options to control datetime rebasing in read

AmplabJenkins removed a comment on pull request #31529: URL: https://github.com/apache/spark/pull/31529#issuecomment-775605752 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39634/

[GitHub] [spark] beliefer commented on a change in pull request #31337: [SPARK-34234][SQL] Remove TreeNodeException that didn't work

beliefer commented on a change in pull request #31337: URL: https://github.com/apache/spark/pull/31337#discussion_r572520513 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala ## @@ -504,8 +504,7 @@ abstract class TreeNode[BaseType <:

[GitHub] [spark] beliefer commented on a change in pull request #31337: [SPARK-34234][SQL] Remove TreeNodeException that didn't work

beliefer commented on a change in pull request #31337: URL: https://github.com/apache/spark/pull/31337#discussion_r572521967 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ## @@ -157,7 +156,7 @@ case class ExplainCommand( overrid

[GitHub] [spark] rednaxelafx commented on pull request #31509: [SPARK-34396][SQL] Add a new build-in function delegate

rednaxelafx commented on pull request #31509: URL: https://github.com/apache/spark/pull/31509#issuecomment-775610985 Just some random thoughts: This kind of functionality is certain useful for debugging and could come in handy for writing shorthands, but I have mixed feelings about e

[GitHub] [spark] SparkQA commented on pull request #31531: [SPARK-34080][ML][PYTHON][FOLLOW-UP] Update score function in document

SparkQA commented on pull request #31531: URL: https://github.com/apache/spark/pull/31531#issuecomment-775611852 **[Test build #135053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135053/testReport)** for PR 31531 at commit [`2ee9913`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.

SparkQA commented on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-775611895 **[Test build #135048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135048/testReport)** for PR 31510 at commit [`bff7afd`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats

SparkQA commented on pull request #31485: URL: https://github.com/apache/spark/pull/31485#issuecomment-775611880 **[Test build #135054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135054/testReport)** for PR 31485 at commit [`b29b2dc`](https://github.com

[GitHub] [spark] SparkQA removed a comment on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.

SparkQA removed a comment on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-775509993 **[Test build #135048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135048/testReport)** for PR 31510 at commit [`bff7afd`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31516: [SPARK-34238][SQL][FOLLOW_UP] SHOW PARTITIONS Keep consistence with other `SHOW` command

SparkQA commented on pull request #31516: URL: https://github.com/apache/spark/pull/31516#issuecomment-775612051 **[Test build #135055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135055/testReport)** for PR 31516 at commit [`43a6c5d`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

SparkQA commented on pull request #31495: URL: https://github.com/apache/spark/pull/31495#issuecomment-775612140 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39635/ ---

[GitHub] [spark] AmplabJenkins commented on pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

AmplabJenkins commented on pull request #31495: URL: https://github.com/apache/spark/pull/31495#issuecomment-775612160 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39635/ -

[GitHub] [spark] wangyum commented on a change in pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats

wangyum commented on a change in pull request #31485: URL: https://github.com/apache/spark/pull/31485#discussion_r572524056 ## File path: sql/core/src/test/resources/sql-tests/results/explain-cbo.sql.out ## @@ -0,0 +1,80 @@ +-- Automatically generated by SQLQueryTestSuite +-- N

[GitHub] [spark] AngersZhuuuu commented on pull request #31520: [SPARK-34355][CORE][SQL][FOLLOWUP] Log commit time in all File Writer

AngersZh commented on pull request #31520: URL: https://github.com/apache/spark/pull/31520#issuecomment-775612366 > I'm sorry I haven't considered about the impact of extracting the code - logger. I'm not sure we are leveraging the information on classname difference for tracking log o

[GitHub] [spark] Ngone51 commented on a change in pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

Ngone51 commented on a change in pull request #31495: URL: https://github.com/apache/spark/pull/31495#discussion_r572524552 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLog.scala ## @@ -46,6 +47,23 @@ import org.apache.spark.sql.conne

[GitHub] [spark] srowen closed pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.

srowen closed pull request #31510: URL: https://github.com/apache/spark/pull/31510 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [spark] srowen commented on pull request #31510: [SPARK-34395][SQL]Clean up unused code for code simplifications.

srowen commented on pull request #31510: URL: https://github.com/apache/spark/pull/31510#issuecomment-775612869 Merged to master This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] SparkQA commented on pull request #31520: [SPARK-34355][CORE][SQL][FOLLOWUP] Log commit time in all File Writer

SparkQA commented on pull request #31520: URL: https://github.com/apache/spark/pull/31520#issuecomment-775616013 **[Test build #135056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135056/testReport)** for PR 31520 at commit [`6c3bdc6`](https://github.com

[GitHub] [spark] sarutak commented on a change in pull request #31442: [SPARK-34333][SQL] Fix PostgresDialect to handle money types properly

sarutak commented on a change in pull request #31442: URL: https://github.com/apache/spark/pull/31442#discussion_r572527343 ## File path: docs/sql-migration-guide.md ## @@ -24,6 +24,8 @@ license: | ## Upgrading from Spark SQL 3.1 to 3.2 + - In Spark 3.2, money type in Pos

[GitHub] [spark] HeartSaVioR commented on a change in pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

HeartSaVioR commented on a change in pull request #31495: URL: https://github.com/apache/spark/pull/31495#discussion_r572532292 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLog.scala ## @@ -46,6 +47,23 @@ import org.apache.spark.sql.c

[GitHub] [spark] zzcclp commented on pull request #31466: [SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run on windows system

zzcclp commented on pull request #31466: URL: https://github.com/apache/spark/pull/31466#issuecomment-775618187 > Merged to master and branch-3.1. @HyukjinKwon after merged this pr into branch-3.1, there is an 'not found: value TestUtils' error in SQLQueryTestSuite, root cause is mis

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31485: [SPARK-34137][SQL] Update suquery's stats when build LogicalPlan's stats

AngersZh commented on a change in pull request #31485: URL: https://github.com/apache/spark/pull/31485#discussion_r572535922 ## File path: sql/core/src/test/resources/sql-tests/results/explain-cbo.sql.out ## @@ -0,0 +1,80 @@ +-- Automatically generated by SQLQueryTestSuite

<    1   2   3   4   5   6   7   8   >