[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r516912542 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,966 @@ +/* + * Licensed to

[GitHub] [spark] venkata91 commented on a change in pull request #30164: [SPARK-32919][SHUFFLE] Driver side changes for coordinating push based shuffle by selecting external shuffle services for mergi

venkata91 commented on a change in pull request #30164: URL: https://github.com/apache/spark/pull/30164#discussion_r516910659 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -657,6 +679,14 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] SparkQA commented on pull request #30164: [SPARK-32919][SHUFFLE] Driver side changes for coordinating push based shuffle by selecting external shuffle services for merging partitions

SparkQA commented on pull request #30164: URL: https://github.com/apache/spark/pull/30164#issuecomment-721334930 **[Test build #130575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130575/testReport)** for PR 30164 at commit

[GitHub] [spark] attilapiros commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

attilapiros commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r516909855 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,966 @@ +/* + *

[GitHub] [spark] SparkQA commented on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

SparkQA commented on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721332722 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35175/

[GitHub] [spark] mridulm commented on a change in pull request #30096: [SPARK-33185][YARN][WIP] Set up yarn.Client to print direct links to driver stdout/stderr

mridulm commented on a change in pull request #30096: URL: https://github.com/apache/spark/pull/30096#discussion_r516895583 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -1172,6 +1181,31 @@ private[spark] class Client(

[GitHub] [spark] venkata91 commented on a change in pull request #30164: [SPARK-32919][SHUFFLE] Driver side changes for coordinating push based shuffle by selecting external shuffle services for mergi

venkata91 commented on a change in pull request #30164: URL: https://github.com/apache/spark/pull/30164#discussion_r516906481 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala ## @@ -161,6 +170,27 @@

[GitHub] [spark] venkata91 commented on a change in pull request #30164: [SPARK-32919][SHUFFLE] Driver side changes for coordinating push based shuffle by selecting external shuffle services for mergi

venkata91 commented on a change in pull request #30164: URL: https://github.com/apache/spark/pull/30164#discussion_r516905738 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1252,6 +1254,28 @@ private[spark] class DAGScheduler(

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

AmplabJenkins removed a comment on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721330296 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

AmplabJenkins commented on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721330296 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

SparkQA removed a comment on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721245566 **[Test build #130570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130570/testReport)** for PR 30232 at commit

[GitHub] [spark] SparkQA commented on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

SparkQA commented on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721329713 **[Test build #130570 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130570/testReport)** for PR 30232 at commit

[GitHub] [spark] imback82 commented on a change in pull request #30079: [SPARK-33174][SQL] Migrate DROP TABLE to use UnresolvedTableOrView to resolve the identifier

imback82 commented on a change in pull request #30079: URL: https://github.com/apache/spark/pull/30079#discussion_r516901187 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala ## @@ -228,8 +228,11 @@ class

[GitHub] [spark] SparkQA commented on pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

SparkQA commented on pull request #27735: URL: https://github.com/apache/spark/pull/27735#issuecomment-721324633 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35174/

[GitHub] [spark] venkata91 commented on a change in pull request #30164: [SPARK-32919][SHUFFLE] Driver side changes for coordinating push based shuffle by selecting external shuffle services for mergi

venkata91 commented on a change in pull request #30164: URL: https://github.com/apache/spark/pull/30164#discussion_r516894315 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -657,6 +679,14 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29247: [SPARK-32446][SHS] Add new executor metrics summary REST APIs

AmplabJenkins removed a comment on pull request #29247: URL: https://github.com/apache/spark/pull/29247#issuecomment-721318769 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] otterc commented on a change in pull request #30096: [SPARK-33185][YARN][WIP] Set up yarn.Client to print direct links to driver stdout/stderr

otterc commented on a change in pull request #30096: URL: https://github.com/apache/spark/pull/30096#discussion_r516891488 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -1172,6 +1181,31 @@ private[spark] class Client(

[GitHub] [spark] AmplabJenkins commented on pull request #29247: [SPARK-32446][SHS] Add new executor metrics summary REST APIs

AmplabJenkins commented on pull request #29247: URL: https://github.com/apache/spark/pull/29247#issuecomment-721318769 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29247: [SPARK-32446][SHS] Add new executor metrics summary REST APIs

SparkQA commented on pull request #29247: URL: https://github.com/apache/spark/pull/29247#issuecomment-721318736 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35173/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

AmplabJenkins removed a comment on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721315102 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

SparkQA commented on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721315080 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35172/

[GitHub] [spark] AmplabJenkins commented on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

AmplabJenkins commented on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721315102 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

SparkQA commented on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721313475 **[Test build #130574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130574/testReport)** for PR 30229 at commit

[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r516884181 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,966 @@ +/* + * Licensed to

[GitHub] [spark] SparkQA commented on pull request #29247: [SPARK-32446][SHS] Add new executor metrics summary REST APIs

SparkQA commented on pull request #29247: URL: https://github.com/apache/spark/pull/29247#issuecomment-721306840 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35173/

[GitHub] [spark] SparkQA commented on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

SparkQA commented on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721304480 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35172/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

AmplabJenkins removed a comment on pull request #27735: URL: https://github.com/apache/spark/pull/27735#issuecomment-721303724 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

AmplabJenkins removed a comment on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721303408 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

SparkQA removed a comment on pull request #27735: URL: https://github.com/apache/spark/pull/27735#issuecomment-721295372 **[Test build #130573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130573/testReport)** for PR 27735 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

AmplabJenkins removed a comment on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721303402 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

AmplabJenkins commented on pull request #27735: URL: https://github.com/apache/spark/pull/27735#issuecomment-721303724 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

SparkQA commented on pull request #27735: URL: https://github.com/apache/spark/pull/27735#issuecomment-721303477 **[Test build #130573 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130573/testReport)** for PR 27735 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

AmplabJenkins commented on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721303402 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

SparkQA removed a comment on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721284185 **[Test build #130572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130572/testReport)** for PR 30229 at commit

[GitHub] [spark] SparkQA commented on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

SparkQA commented on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721303197 **[Test build #130572 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130572/testReport)** for PR 30229 at commit

[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r516871106 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,966 @@ +/* + * Licensed to

[GitHub] [spark] PedroCorreiaLuis commented on pull request #26319: [SPARK-29594][SQL] Provide better error message when creating a Dataset from a Sequence of Case class where a field name started wit

PedroCorreiaLuis commented on pull request #26319: URL: https://github.com/apache/spark/pull/26319#issuecomment-721298978 @kiszk So a couple of things: - I couldn't find any Character.isJavaIdentifier. - if you have the case class Foo that you mentioned it will enter in my

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

dongjoon-hyun commented on a change in pull request #27735: URL: https://github.com/apache/spark/pull/27735#discussion_r516867308 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala ## @@ -64,7 +64,9 @@ private[spark]

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

dongjoon-hyun commented on a change in pull request #27735: URL: https://github.com/apache/spark/pull/27735#discussion_r516867095 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala ## @@ -64,7 +64,9 @@ private[spark]

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

dongjoon-hyun commented on a change in pull request #27735: URL: https://github.com/apache/spark/pull/27735#discussion_r516866765 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala ## @@ -64,7 +64,9 @@ private[spark]

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

dongjoon-hyun commented on a change in pull request #27735: URL: https://github.com/apache/spark/pull/27735#discussion_r516865916 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/Fabric8Aliases.scala ## @@ -16,14 +16,18 @@ */

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

dongjoon-hyun commented on a change in pull request #27735: URL: https://github.com/apache/spark/pull/27735#discussion_r516865507 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/K8sVolumeTestUtils.scala ## @@ -20,7 +20,7 @@ import

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

dongjoon-hyun commented on a change in pull request #27735: URL: https://github.com/apache/spark/pull/27735#discussion_r516865810 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/Fabric8Aliases.scala ## @@ -16,14 +16,18 @@ */

[GitHub] [spark] SparkQA commented on pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

SparkQA commented on pull request #27735: URL: https://github.com/apache/spark/pull/27735#issuecomment-721295372 **[Test build #130573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130573/testReport)** for PR 27735 at commit

[GitHub] [spark] xkrogen commented on a change in pull request #30096: [SPARK-33185][YARN][WIP] Set up yarn.Client to print direct links to driver stdout/stderr

xkrogen commented on a change in pull request #30096: URL: https://github.com/apache/spark/pull/30096#discussion_r516863924 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -1172,6 +1181,31 @@ private[spark] class Client(

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

dongjoon-hyun commented on a change in pull request #27735: URL: https://github.com/apache/spark/pull/27735#discussion_r516863867 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/MountConfDirExecutorFeatureStepSuite.scala ##

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

dongjoon-hyun commented on a change in pull request #27735: URL: https://github.com/apache/spark/pull/27735#discussion_r516863350 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountConfDirExecutorFeatureStep.scala ## @@

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

dongjoon-hyun commented on a change in pull request #27735: URL: https://github.com/apache/spark/pull/27735#discussion_r516863714 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/MountConfDirExecutorFeatureStepSuite.scala ##

[GitHub] [spark] xkrogen commented on a change in pull request #30096: [SPARK-33185][YARN][WIP] Set up yarn.Client to print direct links to driver stdout/stderr

xkrogen commented on a change in pull request #30096: URL: https://github.com/apache/spark/pull/30096#discussion_r516863291 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -1172,6 +1181,31 @@ private[spark] class Client(

[GitHub] [spark] dongjoon-hyun commented on pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

dongjoon-hyun commented on pull request #27735: URL: https://github.com/apache/spark/pull/27735#issuecomment-721292593 Thank you for pinging me, @ScrapCodes . I'll take a look today again. This is an automated message from

[GitHub] [spark] dongjoon-hyun commented on pull request #27735: [SPARK-30985][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

dongjoon-hyun commented on pull request #27735: URL: https://github.com/apache/spark/pull/27735#issuecomment-721292262 Retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] zsxwing edited a comment on pull request #30151: [SPARK-33223][SS][UI]Structured Streaming Web UI state information

zsxwing edited a comment on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-721290476 Thanks for adding this. This is pretty useful. Could you add some simple tests in

[GitHub] [spark] zsxwing commented on pull request #30151: [SPARK-33223][SS][UI]Structured Streaming Web UI state information

zsxwing commented on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-721290476 Thanks for adding this. This is pretty useful. Could you add some simple tests in

[GitHub] [spark] otterc commented on a change in pull request #30096: [SPARK-33185][YARN][WIP] Set up yarn.Client to print direct links to driver stdout/stderr

otterc commented on a change in pull request #30096: URL: https://github.com/apache/spark/pull/30096#discussion_r516858215 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -1172,6 +1181,31 @@ private[spark] class Client(

[GitHub] [spark] dongjoon-hyun closed pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

dongjoon-hyun closed pull request #30232: URL: https://github.com/apache/spark/pull/30232 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] dongjoon-hyun commented on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

dongjoon-hyun commented on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721288282 All GitHub Actions passed. Merged to branch-2.4. This is an automated message from the Apache Git

[GitHub] [spark] dongjoon-hyun commented on pull request #30235: [SPARK-33293][SQL][FOLLOW-UP] Rename TableWriteExec to TableWriteExecHelper

dongjoon-hyun commented on pull request #30235: URL: https://github.com/apache/spark/pull/30235#issuecomment-721286763 cc @holdenk for the K8s IT failure. ``` - Test basic decommissioning with shuffle cleanup *** FAILED *** ```

[GitHub] [spark] imback82 commented on a change in pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

imback82 commented on a change in pull request #30229: URL: https://github.com/apache/spark/pull/30229#discussion_r516853032 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala ## @@ -280,6 +280,9 @@ class

[GitHub] [spark] imback82 commented on a change in pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

imback82 commented on a change in pull request #30229: URL: https://github.com/apache/spark/pull/30229#discussion_r516853691 ## File path: sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala ## @@ -419,17 +410,16 @@ class

[GitHub] [spark] SparkQA commented on pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

SparkQA commented on pull request #30229: URL: https://github.com/apache/spark/pull/30229#issuecomment-721284185 **[Test build #130572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130572/testReport)** for PR 30229 at commit

[GitHub] [spark] srowen commented on a change in pull request #30203: [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls

srowen commented on a change in pull request #30203: URL: https://github.com/apache/spark/pull/30203#discussion_r516851464 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala ## @@ -218,13 +218,22 @@ object ExtractPythonUDFs

[GitHub] [spark] peter-toth commented on pull request #29170: [SPARK-30876][SQL] Optimizer fails to infer constraints within join

peter-toth commented on pull request #29170: URL: https://github.com/apache/spark/pull/29170#issuecomment-721281470 Hmm, I've run into this issue many times recently. Spark 2.3+ runs `InferFiltersFromConstraints` rule only once due to https://github.com/apache/spark/pull/19149, which

[GitHub] [spark] SparkQA commented on pull request #29247: [SPARK-32446][SHS] Add new executor metrics summary REST APIs

SparkQA commented on pull request #29247: URL: https://github.com/apache/spark/pull/29247#issuecomment-721280219 **[Test build #130571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130571/testReport)** for PR 29247 at commit

[GitHub] [spark] holdenk commented on pull request #30206: [SPARK-33261][K8S] Add a developer API for custom feature steps

holdenk commented on pull request #30206: URL: https://github.com/apache/spark/pull/30206#issuecomment-721279189 So since we allow the user to specify templates anyways I don't see this as opening any additional security concerns. As for why templates are not enough, for some deployments

[GitHub] [spark] holdenk commented on a change in pull request #30206: [SPARK-33261][K8S] Add a developer API for custom feature steps

holdenk commented on a change in pull request #30206: URL: https://github.com/apache/spark/pull/30206#discussion_r516844344 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KubernetesFeatureConfigStep.scala ## @@ -18,13

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30235: [SPARK-33293][SQL][FOLLOW-UP] Rename TableWriteExec to TableWriteExecHelper

AmplabJenkins removed a comment on pull request #30235: URL: https://github.com/apache/spark/pull/30235#issuecomment-721275954 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30235: [SPARK-33293][SQL][FOLLOW-UP] Rename TableWriteExec to TableWriteExecHelper

AmplabJenkins removed a comment on pull request #30235: URL: https://github.com/apache/spark/pull/30235#issuecomment-721275945 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #30235: [SPARK-33293][SQL][FOLLOW-UP] Rename TableWriteExec to TableWriteExecHelper

AmplabJenkins commented on pull request #30235: URL: https://github.com/apache/spark/pull/30235#issuecomment-721275945 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30235: [SPARK-33293][SQL][FOLLOW-UP] Rename TableWriteExec to TableWriteExecHelper

SparkQA commented on pull request #30235: URL: https://github.com/apache/spark/pull/30235#issuecomment-721275914 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35170/

[GitHub] [spark] xkrogen commented on pull request #30122: [SPARK-33214][TEST][HIVE] Stop HiveExternalCatalogVersionsSuite from using a hard-coded location to store localized Spark binaries.

xkrogen commented on pull request #30122: URL: https://github.com/apache/spark/pull/30122#issuecomment-721275156 Ping @cloud-fan @gatorsmile who were involved in the original PR. This is an automated message from the Apache

[GitHub] [spark] peter-toth commented on a change in pull request #30203: [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls

peter-toth commented on a change in pull request #30203: URL: https://github.com/apache/spark/pull/30203#discussion_r516836683 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala ## @@ -218,13 +218,22 @@ object ExtractPythonUDFs

[GitHub] [spark] peter-toth commented on a change in pull request #30203: [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls

peter-toth commented on a change in pull request #30203: URL: https://github.com/apache/spark/pull/30203#discussion_r516836683 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala ## @@ -218,13 +218,22 @@ object ExtractPythonUDFs

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

AmplabJenkins removed a comment on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721269486 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] peter-toth commented on a change in pull request #30203: [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls

peter-toth commented on a change in pull request #30203: URL: https://github.com/apache/spark/pull/30203#discussion_r516836683 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala ## @@ -218,13 +218,22 @@ object ExtractPythonUDFs

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

AmplabJenkins removed a comment on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721269476 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

AmplabJenkins commented on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721269476 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

SparkQA commented on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721269441 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35171/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

AmplabJenkins removed a comment on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721262642 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

SparkQA removed a comment on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721156123 **[Test build #130568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130568/testReport)** for PR 30234 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

AmplabJenkins commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721262642 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30235: [SPARK-33293][SQL][FOLLOW-UP] Rename TableWriteExec to TableWriteExecHelper

SparkQA commented on pull request #30235: URL: https://github.com/apache/spark/pull/30235#issuecomment-721262573 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35170/

[GitHub] [spark] SparkQA commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

SparkQA commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721261338 **[Test build #130568 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130568/testReport)** for PR 30234 at commit

[GitHub] [spark] SparkQA commented on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

SparkQA commented on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721260521 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35171/

[GitHub] [spark] dongjoon-hyun commented on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

dongjoon-hyun commented on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721258855 Thank you, @viirya ! This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] viirya edited a comment on pull request #30210: [SPARK-33259][SS] Disable streaming query with possible correctness issue by default

viirya edited a comment on pull request #30210: URL: https://github.com/apache/spark/pull/30210#issuecomment-721250207 Hmm, based on what I saw, it seems to me the discussion on dev@ mailing list is not so active, and the PR attracts more discussion in Spark community, but I'm okay to

[GitHub] [spark] viirya commented on pull request #30210: [SPARK-33259][SS] Disable streaming query with possible correctness issue by default

viirya commented on pull request #30210: URL: https://github.com/apache/spark/pull/30210#issuecomment-721250207 Hmm, based on what I saw, it seems to me the discussion on dev@ mailing list is not so active, but I'm okay to drop some words in dev@ mailing list and see if we can get some

[GitHub] [spark] viirya commented on pull request #26935: [SPARK-30294][SS] Explicitly defines read-only StateStore and optimize for HDFSBackedStateStore

viirya commented on pull request #26935: URL: https://github.com/apache/spark/pull/26935#issuecomment-721247518 Sorry for late. I will do another check today or tomorrow. Thanks. This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

SparkQA commented on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721245566 **[Test build #130570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130570/testReport)** for PR 30232 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #30232: [SPARK-33156][INFRA][2.4] Upgrade GithubAction image from 18.04 to 20.04

dongjoon-hyun commented on pull request #30232: URL: https://github.com/apache/spark/pull/30232#issuecomment-721242725 Thank you, @HyukjinKwon . I removed that line. This is an automated message from the Apache Git Service.

[GitHub] [spark] SparkQA commented on pull request #30235: [SPARK-33293][SQL][FOLLOW-UP] Rename TableWriteExec to TableWriteExecHelper

SparkQA commented on pull request #30235: URL: https://github.com/apache/spark/pull/30235#issuecomment-721232635 **[Test build #130569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130569/testReport)** for PR 30235 at commit

[GitHub] [spark] sunchao opened a new pull request #30235: [SPARK-33293][SQL][FOLLOW-UP] Rename TableWriteExec to TableWriteExecHelper

sunchao opened a new pull request #30235: URL: https://github.com/apache/spark/pull/30235 ### What changes were proposed in this pull request? Rename `TableWriteExec` in `WriteToDataSourceV2Exec.scala` to `TableWriteExecHelper`. ### Why are the changes needed?

[GitHub] [spark] attilapiros commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

attilapiros commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r516776712 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,966 @@ +/* + *

[GitHub] [spark] ahshahid commented on pull request #30185: [SPARK-33152][SQL] This PR proposes a new logic to maintain & track constraints which solves the OOM or performance issues in query compilat

ahshahid commented on pull request #30185: URL: https://github.com/apache/spark/pull/30185#issuecomment-721216214 @srowen Yes, while running the merge with changes locally I accidentally pushed the changes. I will fix it. I have made some more changes which alleviates the need to

[GitHub] [spark] bersprockets commented on a change in pull request #30221: [SPARK-33314][SQL] Avoid dropping rows in Avro reader

bersprockets commented on a change in pull request #30221: URL: https://github.com/apache/spark/pull/30221#discussion_r516771531 ## File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ## @@ -2005,3 +2027,61 @@ class AvroV2Suite extends AvroSuite

[GitHub] [spark] bersprockets commented on pull request #30221: [SPARK-33314][SQL] Avoid dropping rows in Avro reader

bersprockets commented on pull request #30221: URL: https://github.com/apache/spark/pull/30221#issuecomment-721213619 > FYI I just tried and can't find a scenario that has multiple method calls on `hasNext()` without `next()`. @gengliangwang My reprod case is such an example. When

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

AmplabJenkins removed a comment on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721212161 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

AmplabJenkins commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721212161 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30234: [SPARK-33285][CORE][SQL] Fix deprecated compilation warnings of "Auto-application to () is deprecated" in Scala 2.13

SparkQA commented on pull request #30234: URL: https://github.com/apache/spark/pull/30234#issuecomment-721212133 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35169/

[GitHub] [spark] cloud-fan commented on a change in pull request #30229: [SPARK-33321][SQL] Migrate ANALYZE TABLE commands to use UnresolvedTableOrView to resolve the identifier

cloud-fan commented on a change in pull request #30229: URL: https://github.com/apache/spark/pull/30229#discussion_r516763043 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzePartitionCommand.scala ## @@ -75,6 +75,9 @@ case class

[GitHub] [spark] jkleckner commented on pull request #29533: [SPARK-24266][K8S][3.0] Restart the watcher when we receive a version changed from k8s

jkleckner commented on pull request #29533: URL: https://github.com/apache/spark/pull/29533#issuecomment-721206309 @dongjoon-hyun Thank you for the merge and hanging in there. Hopefully @shockdm will revive the 2.4 branch fix kicked off in #29496

<    1   2   3   4   5   6   7   >