[GitHub] [spark] attilapiros commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

attilapiros commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-638684678 > So I don't want to stop the executor directly once the block migration is done. Instead, I have a follow-up JIRA which I've started working on that shutdowns the executor

[GitHub] [spark] xuanyuanking commented on pull request #28707: [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store

xuanyuanking commented on pull request #28707: URL: https://github.com/apache/spark/pull/28707#issuecomment-638679010 @skambha You can check the integrated tests in #28725. If we delete the validation, we'll get a NPE for [this

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27983: [SPARK-15694][SQL]Implement ScriptTransformation in sql/core

AmplabJenkins removed a comment on pull request #27983: URL: https://github.com/apache/spark/pull/27983#issuecomment-638673996 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format

AmplabJenkins commented on pull request #28725: URL: https://github.com/apache/spark/pull/28725#issuecomment-638673832 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format

AmplabJenkins removed a comment on pull request #28725: URL: https://github.com/apache/spark/pull/28725#issuecomment-638673832 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #27983: [SPARK-15694][SQL]Implement ScriptTransformation in sql/core

AmplabJenkins commented on pull request #27983: URL: https://github.com/apache/spark/pull/27983#issuecomment-638673996 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #27983: [SPARK-15694][SQL]Implement ScriptTransformation in sql/core

SparkQA commented on pull request #27983: URL: https://github.com/apache/spark/pull/27983#issuecomment-638673153 **[Test build #123525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123525/testReport)** for PR 27983 at commit

[GitHub] [spark] SparkQA commented on pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format

SparkQA commented on pull request #28725: URL: https://github.com/apache/spark/pull/28725#issuecomment-638673339 **[Test build #123524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123524/testReport)** for PR 28725 at commit

[GitHub] [spark] AngersZhuuuu commented on pull request #27983: [SPARK-15694][SQL]Implement ScriptTransformation in sql/core

AngersZh commented on pull request #27983: URL: https://github.com/apache/spark/pull/27983#issuecomment-638672897 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] xuanyuanking opened a new pull request #28725: [WIP][SPARK-31905][SS] Add compatibility tests for streaming aggregation state store format

xuanyuanking opened a new pull request #28725: URL: https://github.com/apache/spark/pull/28725 ### What changes were proposed in this pull request? Add compatibility tests for streaming aggregation state store format. ### Why are the changes needed? After SPARK-31894, we have a

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28724: [SPARK-31904][SQL] Fix case sensitive problem of char and varchar partition columns

AmplabJenkins removed a comment on pull request #28724: URL: https://github.com/apache/spark/pull/28724#issuecomment-638665168 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28724: [SPARK-31904][SQL] Fix case sensitive problem of char and varchar partition columns

AmplabJenkins commented on pull request #28724: URL: https://github.com/apache/spark/pull/28724#issuecomment-638665168 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] viirya edited a comment on pull request #28662: [SPARK-31850][SQL]Prevent DetermineTableStats from computing stats multiple times for same table

viirya edited a comment on pull request #28662: URL: https://github.com/apache/spark/pull/28662#issuecomment-638663484 I see, it is because calling `executeSameContext` to analyze one logical plan during analysis. I'd say that the reproducing steps in the description is confusing because

[GitHub] [spark] SparkQA commented on pull request #28724: [SPARK-31904][SQL] Fix case sensitive problem of char and varchar partition columns

SparkQA commented on pull request #28724: URL: https://github.com/apache/spark/pull/28724#issuecomment-638664480 **[Test build #123523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123523/testReport)** for PR 28724 at commit

[GitHub] [spark] viirya commented on pull request #28662: [SPARK-31850][SQL]Prevent DetermineTableStats from computing stats multiple times for same table

viirya commented on pull request #28662: URL: https://github.com/apache/spark/pull/28662#issuecomment-638663484 I see, it is because calling `executeSameContext` to analyze one logical plan during analysis. I'd say that the reproducing steps in the description is confusing because it

[GitHub] [spark] LantaoJin opened a new pull request #28724: [SPARK-31904][SQL] Fix case sensitive problem of char and varchar partition columns

LantaoJin opened a new pull request #28724: URL: https://github.com/apache/spark/pull/28724 ### What changes were proposed in this pull request? ```sql CREATE TABLE t1(a STRING, B VARCHAR(10), C CHAR(10)) STORED AS parquet; CREATE TABLE t2 USING parquet PARTITIONED BY (b, c) AS

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0

AmplabJenkins removed a comment on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-638660888 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0

AmplabJenkins commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-638660888 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0

SparkQA commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-638660123 **[Test build #123522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123522/testReport)** for PR 28585 at commit

[GitHub] [spark] sarutak commented on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0

sarutak commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-638657246 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] maropu removed a comment on pull request #28686: [SPARK-31877][SQL]Avoid stats computation for Hive table

maropu removed a comment on pull request #28686: URL: https://github.com/apache/spark/pull/28686#issuecomment-638654746 > Updated statistics will be propagated into HadoopFsRelation and so the sizeInBytes depends on it. Ah, I see. I missed that code flow.

[GitHub] [spark] maropu commented on a change in pull request #28686: [SPARK-31877][SQL]Avoid stats computation for Hive table

maropu commented on a change in pull request #28686: URL: https://github.com/apache/spark/pull/28686#discussion_r435042559 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionStateBuilder.scala ## @@ -82,8 +82,8 @@ class HiveSessionStateBuilder(session:

[GitHub] [spark] maropu commented on pull request #28686: [SPARK-31877][SQL]Avoid stats computation for Hive table

maropu commented on pull request #28686: URL: https://github.com/apache/spark/pull/28686#issuecomment-638654746 > Updated statistics will be propagated into HadoopFsRelation and so the sizeInBytes depends on it. Ah, I see. I missed that code flow.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

AmplabJenkins removed a comment on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638647669 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27983: [SPARK-15694][SQL]Implement ScriptTransformation in sql/core

AmplabJenkins removed a comment on pull request #27983: URL: https://github.com/apache/spark/pull/27983#issuecomment-638647937 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

AmplabJenkins removed a comment on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638647662 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0

AmplabJenkins removed a comment on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-638647688 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #27983: [SPARK-15694][SQL]Implement ScriptTransformation in sql/core

SparkQA removed a comment on pull request #27983: URL: https://github.com/apache/spark/pull/27983#issuecomment-638586972 **[Test build #123517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123517/testReport)** for PR 27983 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0

AmplabJenkins removed a comment on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-638647674 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

SparkQA removed a comment on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638632450 **[Test build #123521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123521/testReport)** for PR 28412 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27983: [SPARK-15694][SQL]Implement ScriptTransformation in sql/core

AmplabJenkins removed a comment on pull request #27983: URL: https://github.com/apache/spark/pull/27983#issuecomment-638647930 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0

SparkQA removed a comment on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-638595499 **[Test build #123518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123518/testReport)** for PR 28585 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0

AmplabJenkins commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-638647674 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #27983: [SPARK-15694][SQL]Implement ScriptTransformation in sql/core

AmplabJenkins commented on pull request #27983: URL: https://github.com/apache/spark/pull/27983#issuecomment-638647930 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

SparkQA commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638647577 **[Test build #123521 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123521/testReport)** for PR 28412 at commit

[GitHub] [spark] SparkQA commented on pull request #28585: [SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0

SparkQA commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-638647578 **[Test build #123518 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123518/testReport)** for PR 28585 at commit

[GitHub] [spark] SparkQA commented on pull request #27983: [SPARK-15694][SQL]Implement ScriptTransformation in sql/core

SparkQA commented on pull request #27983: URL: https://github.com/apache/spark/pull/27983#issuecomment-638647576 **[Test build #123517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123517/testReport)** for PR 27983 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

AmplabJenkins commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638647662 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HeartSaVioR commented on pull request #28707: [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store

HeartSaVioR commented on pull request #28707: URL: https://github.com/apache/spark/pull/28707#issuecomment-638644781 > What are other stateful operations that use unsafe row? I think we can apply the check everywhere. State store itself stores UnsafeRow, hence it applies to

[GitHub] [spark] maropu commented on a change in pull request #28490: [SPARK-31670][SQL]Struct Field in groupByExpr with CUBE

maropu commented on a change in pull request #28490: URL: https://github.com/apache/spark/pull/28490#discussion_r435029537 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -506,31 +506,55 @@ class Analyzer(

[GitHub] [spark] karuppayya commented on pull request #28715: [SPARK-31897][SQL]Enable codegen for GenerateExec

karuppayya commented on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-638637169 @dongjoon-hyun @gatorsmile @cloud-fan can anyone help review this change? Thanks This is an automated

[GitHub] [spark] HyukjinKwon commented on pull request #28723: [SPARK-28624][SQL][TESTS][3.0] Run date.sql via Thrift Server

HyukjinKwon commented on pull request #28723: URL: https://github.com/apache/spark/pull/28723#issuecomment-638634561 Let's cut out RC3 first and then merge. This is an automated message from the Apache Git Service. To

[GitHub] [spark] wangyum commented on a change in pull request #28575: [SPARK-31705][SQL] Push predicate through join by rewriting join condition to conjunctive normal form

wangyum commented on a change in pull request #28575: URL: https://github.com/apache/spark/pull/28575#discussion_r435021535 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -1372,6 +1377,96 @@ object

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

AmplabJenkins removed a comment on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638632997 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

AmplabJenkins commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638632997 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

SparkQA commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638632450 **[Test build #123521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123521/testReport)** for PR 28412 at commit

[GitHub] [spark] karuppayya commented on a change in pull request #28662: [SPARK-31850][SQL]Prevent DetermineTableStats from computing stats multiple times for same table

karuppayya commented on a change in pull request #28662: URL: https://github.com/apache/spark/pull/28662#discussion_r435019904 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala ## @@ -112,18 +112,20 @@ class ResolveHiveSerdeTable(session:

[GitHub] [spark] karuppayya commented on a change in pull request #28662: [SPARK-31850][SQL]Prevent DetermineTableStats from computing stats multiple times for same table

karuppayya commented on a change in pull request #28662: URL: https://github.com/apache/spark/pull/28662#discussion_r435019316 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ## @@ -851,12 +851,21 @@ object DDLUtils {

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #28575: [SPARK-31705][SQL] Push predicate through join by rewriting join condition to conjunctive normal form

AngersZh commented on a change in pull request #28575: URL: https://github.com/apache/spark/pull/28575#discussion_r435018473 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -1372,6 +1377,96 @@ object

[GitHub] [spark] karuppayya edited a comment on pull request #28662: [SPARK-31850][SQL]Prevent DetermineTableStats from computing stats multiple times for same table

karuppayya edited a comment on pull request #28662: URL: https://github.com/apache/spark/pull/28662#issuecomment-638629109 > Can you explain why DetermineTableStats will calculate the statistics multiple times? ``` at

[GitHub] [spark] karuppayya commented on pull request #28662: [SPARK-31850][SQL]Prevent DetermineTableStats from computing stats multiple times for same table

karuppayya commented on pull request #28662: URL: https://github.com/apache/spark/pull/28662#issuecomment-638629109 > Can you explain why DetermineTableStats will calculate the statistics multiple times? ``` at

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

AmplabJenkins removed a comment on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638627353 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

AmplabJenkins commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638627349 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

AmplabJenkins removed a comment on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638627240 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

SparkQA commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638627332 **[Test build #123520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123520/testReport)** for PR 28412 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

SparkQA removed a comment on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638626777 **[Test build #123520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123520/testReport)** for PR 28412 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

AmplabJenkins commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638627240 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on pull request #28707: [SPARK-31894][SS] Introduce UnsafeRow format validation for streaming state store

cloud-fan commented on pull request #28707: URL: https://github.com/apache/spark/pull/28707#issuecomment-638626662 Yea we need this PR to unblock backporting SPARK-28067 to 3.0. > the mechanism is a validation of the UnsafeRow which can be applied to all stateful operations.

[GitHub] [spark] SparkQA commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

SparkQA commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638626777 **[Test build #123520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123520/testReport)** for PR 28412 at commit

[GitHub] [spark] uncleGen commented on pull request #28391: [SPARK-31593][SS] Remove unnecessary streaming query progress update

uncleGen commented on pull request #28391: URL: https://github.com/apache/spark/pull/28391#issuecomment-638625572 @HeartSaVioR @xuanyuanking Thanks for reviewing. This is an automated message from the Apache Git Service. To

[GitHub] [spark] jiangxb1987 commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully

jiangxb1987 commented on a change in pull request #28641: URL: https://github.com/apache/spark/pull/28641#discussion_r435012863 ## File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ## @@ -889,16 +870,40 @@ class DAGSchedulerSuite extends

[GitHub] [spark] karuppayya commented on a change in pull request #28686: [SPARK-31877][SQL]Avoid stats computation for Hive table

karuppayya commented on a change in pull request #28686: URL: https://github.com/apache/spark/pull/28686#discussion_r435010420 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionStateBuilder.scala ## @@ -82,8 +82,8 @@ class

<    1   2   3   4