[GitHub] [spark] gatorsmile commented on pull request #29630: [SPARK-32097] Enable Spark History Server to read from multiple directories

gatorsmile commented on pull request #29630: URL: https://github.com/apache/spark/pull/29630#issuecomment-687553724 cc @rednaxelafx @jiangxb1987 This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] gatorsmile commented on pull request #29638: [SPARK-32687][SQL] Let CostBasedJoinReorder produce relatively deterministic optimization result

gatorsmile commented on pull request #29638: URL: https://github.com/apache/spark/pull/29638#issuecomment-687553589 cc @maryannxue @cloud-fan This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] SparkQA commented on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

SparkQA commented on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687529061 **[Test build #128316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128316/testReport)** for PR 29652 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

AmplabJenkins commented on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687529159 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

AmplabJenkins removed a comment on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687529159 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

SparkQA removed a comment on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687523835 **[Test build #128316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128316/testReport)** for PR 29652 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

SparkQA removed a comment on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-687488484 **[Test build #128314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128314/testReport)** for PR 29645 at commit

[GitHub] [spark] SparkQA commented on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

SparkQA commented on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-687548622 **[Test build #128314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128314/testReport)** for PR 29645 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

AmplabJenkins commented on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687521797 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

AmplabJenkins removed a comment on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687521797 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

SparkQA removed a comment on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687521350 **[Test build #128315 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128315/testReport)** for PR 29652 at commit

[GitHub] [spark] SparkQA commented on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

SparkQA commented on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687521787 **[Test build #128315 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128315/testReport)** for PR 29652 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

AmplabJenkins removed a comment on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687521802 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

SparkQA removed a comment on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-687477282 **[Test build #128313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128313/testReport)** for PR 29645 at commit

[GitHub] [spark] SparkQA commented on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

SparkQA commented on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-687545413 **[Test build #128313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128313/testReport)** for PR 29645 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

AmplabJenkins removed a comment on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-687545627 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

AmplabJenkins commented on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-687545627 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29605: [SPARK-31511][SQL][2.4] Make BytesToBytesMap iterators thread-safe

SparkQA commented on pull request #29605: URL: https://github.com/apache/spark/pull/29605#issuecomment-687556780 **[Test build #128317 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128317/testReport)** for PR 29605 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29605: [SPARK-31511][SQL][2.4] Make BytesToBytesMap iterators thread-safe

AmplabJenkins commented on pull request #29605: URL: https://github.com/apache/spark/pull/29605#issuecomment-687556917 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

SparkQA commented on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687521350 **[Test build #128315 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128315/testReport)** for PR 29652 at commit

[GitHub] [spark] github-actions[bot] closed pull request #28581: [SPARK-31236][DSTREAMS][Kinesis] KCL 2 support added to solve few ongoing issue with KCL 1 implementation

github-actions[bot] closed pull request #28581: URL: https://github.com/apache/spark/pull/28581 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] ulysses-you opened a new pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

ulysses-you opened a new pull request #29652: URL: https://github.com/apache/spark/pull/29652 ### What changes were proposed in this pull request? Cache `InterruptedException` error and throw a `SparkException` in `SparkRackResolver`. ### Why are the changes

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

AmplabJenkins removed a comment on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-687548933 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

AmplabJenkins commented on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-687548933 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

AmplabJenkins commented on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687524244 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

AmplabJenkins removed a comment on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687524244 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29652: [SPARK-32803][CORE] Catch InterruptedException when resolve rack in SparkRackResolver

SparkQA commented on pull request #29652: URL: https://github.com/apache/spark/pull/29652#issuecomment-687523835 **[Test build #128316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128316/testReport)** for PR 29652 at commit

[GitHub] [spark] gatorsmile commented on pull request #29629: [SPARK-32135] Show Spark Driver name on Spark history web page

gatorsmile commented on pull request #29629: URL: https://github.com/apache/spark/pull/29629#issuecomment-687552574 cc @gengliangwang This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] gatorsmile commented on a change in pull request #28996: [SPARK-29358][SQL] Make unionByName optionally fill missing columns with nulls

gatorsmile commented on a change in pull request #28996: URL: https://github.com/apache/spark/pull/28996#discussion_r483418810 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ## @@ -2030,7 +2030,47 @@ class Dataset[T] private[sql]( * @group

[GitHub] [spark] SparkQA commented on pull request #29087: [SPARK-28227][SQL] Support TRANSFORM with aggregation

SparkQA commented on pull request #29087: URL: https://github.com/apache/spark/pull/29087#issuecomment-686955890 **[Test build #128276 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128276/testReport)** for PR 29087 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29643: [SPARK-32638][SQL][FOLLOWUP] Move the plan rewriting methods to QueryPlan

AmplabJenkins removed a comment on pull request #29643: URL: https://github.com/apache/spark/pull/29643#issuecomment-686962350 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] LuciferYang edited a comment on pull request #29638: [SPARK-32687][SQL] Let CostBasedJoinReorder produce relatively deterministic optimization result

LuciferYang edited a comment on pull request #29638: URL: https://github.com/apache/spark/pull/29638#issuecomment-686890566 > Hmm, why this is needed? Firstly I thought CostBasedJoinReorder will produce non-deterministic for same query. But I looked at the JIRA description, seems for

[GitHub] [spark] AmplabJenkins commented on pull request #29643: [SPARK-32638][SQL][FOLLOWUP] Move the plan rewriting methods to QueryPlan

AmplabJenkins commented on pull request #29643: URL: https://github.com/apache/spark/pull/29643#issuecomment-686962350 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] LuciferYang removed a comment on pull request #29638: [SPARK-32687][SQL] Let CostBasedJoinReorder produce relatively deterministic optimization result

LuciferYang removed a comment on pull request #29638: URL: https://github.com/apache/spark/pull/29638#issuecomment-686899631 @viirya I'm also entangled in this issue :( This is an automated message from the Apache Git

[GitHub] [spark] yaooqinn commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null

yaooqinn commented on pull request #29635: URL: https://github.com/apache/spark/pull/29635#issuecomment-686982439 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] yaooqinn removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null

yaooqinn removed a comment on pull request #29635: URL: https://github.com/apache/spark/pull/29635#issuecomment-686982450 OK, are you targeting this to 3.0 or not? This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

AmplabJenkins removed a comment on pull request #29634: URL: https://github.com/apache/spark/pull/29634#issuecomment-686937104 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29646: [SPARK-XXXXX][SQL][TEST] Add tests to check if since fields are set correctly in ExpressionInfo

SparkQA commented on pull request #29646: URL: https://github.com/apache/spark/pull/29646#issuecomment-686946728 **[Test build #128280 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128280/testReport)** for PR 29646 at commit

[GitHub] [spark] maropu commented on pull request #29646: [SPARK-XXXXX][SQL][TEST] Add tests to check if since fields are set correctly in ExpressionInfo

maropu commented on pull request #29646: URL: https://github.com/apache/spark/pull/29646#issuecomment-686945795 NOTE: I will file jira later if necessary. This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] SparkQA commented on pull request #29643: [SPARK-32638][SQL][FOLLOWUP] Move the plan rewriting methods to QueryPlan

SparkQA commented on pull request #29643: URL: https://github.com/apache/spark/pull/29643#issuecomment-686950450 **[Test build #128281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128281/testReport)** for PR 29643 at commit

[GitHub] [spark] HeartSaVioR commented on a change in pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

HeartSaVioR commented on a change in pull request #29461: URL: https://github.com/apache/spark/pull/29461#discussion_r483420929 ## File path: docs/structured-streaming-programming-guide.md ## @@ -861,6 +861,10 @@ isStreaming(df) +You may want to check the logical plan of

[GitHub] [spark] HeartSaVioR commented on a change in pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

HeartSaVioR commented on a change in pull request #29461: URL: https://github.com/apache/spark/pull/29461#discussion_r483420929 ## File path: docs/structured-streaming-programming-guide.md ## @@ -861,6 +861,10 @@ isStreaming(df) +You may want to check the logical plan of

[GitHub] [spark] HeartSaVioR commented on pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

HeartSaVioR commented on pull request #29461: URL: https://github.com/apache/spark/pull/29461#issuecomment-686963445 retest this, please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] cloud-fan commented on a change in pull request #29502: [SPARK-32677][SQL] Load function resource before create

cloud-fan commented on a change in pull request #29502: URL: https://github.com/apache/spark/pull/29502#discussion_r483442972 ## File path: sql/core/src/test/resources/sql-tests/results/udf/udf-udaf.sql.out ## @@ -51,7 +52,7 @@ SELECT default.udaf1(udf(int_col1)) as udaf1,

[GitHub] [spark] SparkQA commented on pull request #29087: [SPARK-28227][SQL] Support TRANSFORM with aggregation

SparkQA commented on pull request #29087: URL: https://github.com/apache/spark/pull/29087#issuecomment-686990484 **[Test build #128287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128287/testReport)** for PR 29087 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29087: [SPARK-28227][SQL] Support TRANSFORM with aggregation

AmplabJenkins commented on pull request #29087: URL: https://github.com/apache/spark/pull/29087#issuecomment-686992750 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

SparkQA removed a comment on pull request #29634: URL: https://github.com/apache/spark/pull/29634#issuecomment-686925219 **[Test build #128277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128277/testReport)** for PR 29634 at commit

[GitHub] [spark] HeartSaVioR commented on a change in pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

HeartSaVioR commented on a change in pull request #29461: URL: https://github.com/apache/spark/pull/29461#discussion_r483410078 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ## @@ -2525,14 +2525,19 @@ class Dataset[T] private[sql]( /** *

[GitHub] [spark] AmplabJenkins commented on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

AmplabJenkins commented on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-686936496 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

AmplabJenkins commented on pull request #29634: URL: https://github.com/apache/spark/pull/29634#issuecomment-686937104 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

SparkQA commented on pull request #29634: URL: https://github.com/apache/spark/pull/29634#issuecomment-686936689 **[Test build #128277 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128277/testReport)** for PR 29634 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

AmplabJenkins removed a comment on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-686936496 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

SparkQA commented on pull request #29461: URL: https://github.com/apache/spark/pull/29461#issuecomment-686943331 **[Test build #128279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128279/testReport)** for PR 29461 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29646: [SPARK-XXXXX][SQL][TEST] Add tests to check if since fields are set correctly in ExpressionInfo

AmplabJenkins commented on pull request #29646: URL: https://github.com/apache/spark/pull/29646#issuecomment-686947407 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29646: [SPARK-XXXXX][SQL][TEST] Add tests to check if since fields are set correctly in ExpressionInfo

AmplabJenkins removed a comment on pull request #29646: URL: https://github.com/apache/spark/pull/29646#issuecomment-686947407 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] gatorsmile commented on a change in pull request #28996: [SPARK-29358][SQL] Make unionByName optionally fill missing columns with nulls

gatorsmile commented on a change in pull request #28996: URL: https://github.com/apache/spark/pull/28996#discussion_r483418373 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ## @@ -2030,7 +2030,47 @@ class Dataset[T] private[sql]( * @group

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29643: [SPARK-32638][SQL][FOLLOWUP] Move the plan rewriting methods to QueryPlan

AmplabJenkins removed a comment on pull request #29643: URL: https://github.com/apache/spark/pull/29643#issuecomment-686951102 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29643: [SPARK-32638][SQL][FOLLOWUP] Move the plan rewriting methods to QueryPlan

AmplabJenkins commented on pull request #29643: URL: https://github.com/apache/spark/pull/29643#issuecomment-686951102 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29087: [SPARK-28227][SQL] Support TRANSFORM with aggregation

AmplabJenkins removed a comment on pull request #29087: URL: https://github.com/apache/spark/pull/29087#issuecomment-686956309 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #29643: [SPARK-32638][SQL][FOLLOWUP] Move the plan rewriting methods to QueryPlan

SparkQA commented on pull request #29643: URL: https://github.com/apache/spark/pull/29643#issuecomment-686961655 **[Test build #128284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128284/testReport)** for PR 29643 at commit

[GitHub] [spark] Fokko commented on a change in pull request #29180: [SPARK-17333][PYSPARK] Enable mypy on the repository

Fokko commented on a change in pull request #29180: URL: https://github.com/apache/spark/pull/29180#discussion_r483431478 ## File path: dev/lint-python ## @@ -122,6 +123,32 @@ function pycodestyle_test { fi } +function mypy_test { +local MYPY_REPORT= +local

[GitHub] [spark] cloud-fan commented on a change in pull request #29589: [SPARK-32748][SQL] Support local property propagation in SubqueryBroadcastExec

cloud-fan commented on a change in pull request #29589: URL: https://github.com/apache/spark/pull/29589#discussion_r483448021 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala ## @@ -1342,6 +1345,52 @@ abstract class

[GitHub] [spark] cloud-fan commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null

cloud-fan commented on pull request #29635: URL: https://github.com/apache/spark/pull/29635#issuecomment-686981169 can we add a migration guide? This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] leanken commented on a change in pull request #29455: [SPARK-32644][SQL] NAAJ support for ShuffleHashJoin when AQE is on

leanken commented on a change in pull request #29455: URL: https://github.com/apache/spark/pull/29455#discussion_r483464874 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/EliminateNullAwareAntiJoin.scala ## @@ -20,22 +20,38 @@ package

[GitHub] [spark] wzhfy commented on a change in pull request #29589: [SPARK-32748][SQL] Support local property propagation in SubqueryBroadcastExec

wzhfy commented on a change in pull request #29589: URL: https://github.com/apache/spark/pull/29589#discussion_r483406090 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala ## @@ -1342,6 +1345,52 @@ abstract class

[GitHub] [spark] xuanyuanking commented on a change in pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

xuanyuanking commented on a change in pull request #29461: URL: https://github.com/apache/spark/pull/29461#discussion_r483416180 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ## @@ -2525,14 +2525,19 @@ class Dataset[T] private[sql]( /** *

[GitHub] [spark] maropu opened a new pull request #29646: [SPARK-XXXXX][SQL][TEST] Add tests to check if since fields are set correctly in ExpressionInfo

maropu opened a new pull request #29646: URL: https://github.com/apache/spark/pull/29646 ### What changes were proposed in this pull request? This PR intends to add a test to check if `since` fields are set correctly in `ExpressionInfo`. This comes from the discussion in

[GitHub] [spark] maropu commented on pull request #29646: [SPARK-XXXXX][SQL][TEST] Add tests to check if since fields are set correctly in ExpressionInfo

maropu commented on pull request #29646: URL: https://github.com/apache/spark/pull/29646#issuecomment-686945464 I don't have much time to check the versions (SPARK-32780) expr-by-expr now, but I think its worth adding the test to prevent one from forgetting setting a since field when

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

AmplabJenkins removed a comment on pull request #29461: URL: https://github.com/apache/spark/pull/29461#issuecomment-686958350 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

AmplabJenkins removed a comment on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-686958416 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29643: [SPARK-32638][SQL][FOLLOWUP] Move the plan rewriting methods to QueryPlan

AmplabJenkins removed a comment on pull request #29643: URL: https://github.com/apache/spark/pull/29643#issuecomment-686958305 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] cloud-fan commented on pull request #29643: [SPARK-32638][SQL][FOLLOWUP] Move the plan rewriting methods to QueryPlan

cloud-fan commented on pull request #29643: URL: https://github.com/apache/spark/pull/29643#issuecomment-686960517 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29646: [SPARK-XXXXX][SQL][TEST] Add tests to check if since fields are set correctly in ExpressionInfo

HyukjinKwon commented on a change in pull request #29646: URL: https://github.com/apache/spark/pull/29646#discussion_r483460277 ## File path: sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala ## @@ -191,4 +191,85 @@ class ExpressionInfoSuite

[GitHub] [spark] HyukjinKwon commented on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

HyukjinKwon commented on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-686997015 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] wzhfy commented on a change in pull request #29589: [SPARK-32748][SQL] Support local property propagation in SubqueryBroadcastExec

wzhfy commented on a change in pull request #29589: URL: https://github.com/apache/spark/pull/29589#discussion_r483404918 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala ## @@ -1342,6 +1345,52 @@ abstract class

[GitHub] [spark] viirya opened a new pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

viirya opened a new pull request #29645: URL: https://github.com/apache/spark/pull/29645 ### What changes were proposed in this pull request? This patch adds nested struct support to `Column.withField` API. ### Why are the changes needed? Currently

[GitHub] [spark] viirya commented on a change in pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

viirya commented on a change in pull request #29645: URL: https://github.com/apache/spark/pull/29645#discussion_r483408113 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithFields.scala ## @@ -0,0 +1,38 @@ +/* + * Licensed to the

[GitHub] [spark] xuanyuanking commented on a change in pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

xuanyuanking commented on a change in pull request #29461: URL: https://github.com/apache/spark/pull/29461#discussion_r483414767 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ## @@ -2525,14 +2525,19 @@ class Dataset[T] private[sql]( /** *

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

AmplabJenkins removed a comment on pull request #29461: URL: https://github.com/apache/spark/pull/29461#issuecomment-686943825 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

AmplabJenkins commented on pull request #29461: URL: https://github.com/apache/spark/pull/29461#issuecomment-686943825 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29087: [SPARK-28227][SQL] Support TRANSFORM with aggregation

AmplabJenkins removed a comment on pull request #29087: URL: https://github.com/apache/spark/pull/29087#issuecomment-686956300 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] manuzhang commented on a change in pull request #29593: [SPARK-32753][SQL] Only copy tags to node with no tags

manuzhang commented on a change in pull request #29593: URL: https://github.com/apache/spark/pull/29593#discussion_r483426598 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala ## @@ -91,7 +91,9 @@ abstract class TreeNode[BaseType <:

[GitHub] [spark] AmplabJenkins commented on pull request #29087: [SPARK-28227][SQL] Support TRANSFORM with aggregation

AmplabJenkins commented on pull request #29087: URL: https://github.com/apache/spark/pull/29087#issuecomment-686956300 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #29087: [SPARK-28227][SQL] Support TRANSFORM with aggregation

SparkQA removed a comment on pull request #29087: URL: https://github.com/apache/spark/pull/29087#issuecomment-686897991 **[Test build #128276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128276/testReport)** for PR 29087 at commit

[GitHub] [spark] HeartSaVioR commented on a change in pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

HeartSaVioR commented on a change in pull request #29461: URL: https://github.com/apache/spark/pull/29461#discussion_r483430511 ## File path: docs/structured-streaming-programming-guide.md ## @@ -861,6 +861,10 @@ isStreaming(df) +You may want to check the logical plan of

[GitHub] [spark] cloud-fan commented on a change in pull request #29643: [SPARK-32638][SQL][FOLLOWUP] Move the plan rewriting methods to QueryPlan

cloud-fan commented on a change in pull request #29643: URL: https://github.com/apache/spark/pull/29643#discussion_r483430462 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala ## @@ -168,6 +170,85 @@ abstract class

[GitHub] [spark] HeartSaVioR commented on a change in pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

HeartSaVioR commented on a change in pull request #29461: URL: https://github.com/apache/spark/pull/29461#discussion_r483430511 ## File path: docs/structured-streaming-programming-guide.md ## @@ -861,6 +861,10 @@ isStreaming(df) +You may want to check the logical plan of

[GitHub] [spark] HeartSaVioR commented on a change in pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

HeartSaVioR commented on a change in pull request #29461: URL: https://github.com/apache/spark/pull/29461#discussion_r483430511 ## File path: docs/structured-streaming-programming-guide.md ## @@ -861,6 +861,10 @@ isStreaming(df) +You may want to check the logical plan of

[GitHub] [spark] AmplabJenkins commented on pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

AmplabJenkins commented on pull request #29461: URL: https://github.com/apache/spark/pull/29461#issuecomment-686965892 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29626: [SPARK-32777][SQL] Aggregation support aggregate function with multiple foldable expressions.

SparkQA commented on pull request #29626: URL: https://github.com/apache/spark/pull/29626#issuecomment-686965358 **[Test build #128285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128285/testReport)** for PR 29626 at commit

[GitHub] [spark] SparkQA commented on pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

SparkQA commented on pull request #29461: URL: https://github.com/apache/spark/pull/29461#issuecomment-686965391 **[Test build #128286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128286/testReport)** for PR 29461 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29626: [SPARK-32777][SQL] Aggregation support aggregate function with multiple foldable expressions.

AmplabJenkins commented on pull request #29626: URL: https://github.com/apache/spark/pull/29626#issuecomment-686965915 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29626: [SPARK-32777][SQL] Aggregation support aggregate function with multiple foldable expressions.

AmplabJenkins removed a comment on pull request #29626: URL: https://github.com/apache/spark/pull/29626#issuecomment-686965915 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29461: [SPARK-32456][SS][FOLLOWUP] Update doc to note about using SQL statement with streaming Dataset

AmplabJenkins removed a comment on pull request #29461: URL: https://github.com/apache/spark/pull/29461#issuecomment-686965892 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29087: [SPARK-28227][SQL] Support TRANSFORM with aggregation

AmplabJenkins removed a comment on pull request #29087: URL: https://github.com/apache/spark/pull/29087#issuecomment-686992750 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

SparkQA commented on pull request #29645: URL: https://github.com/apache/spark/pull/29645#issuecomment-686935891 **[Test build #128278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128278/testReport)** for PR 29645 at commit

[GitHub] [spark] viirya commented on a change in pull request #29645: [SPARK-32796][SQL] Make withField API support nested struct in array

viirya commented on a change in pull request #29645: URL: https://github.com/apache/spark/pull/29645#discussion_r483408113 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithFields.scala ## @@ -0,0 +1,38 @@ +/* + * Licensed to the

[GitHub] [spark] viirya commented on a change in pull request #28996: [SPARK-29358][SQL] Make unionByName optionally fill missing columns with nulls

viirya commented on a change in pull request #28996: URL: https://github.com/apache/spark/pull/28996#discussion_r483419615 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ## @@ -2030,7 +2030,47 @@ class Dataset[T] private[sql]( * @group typedrel

[GitHub] [spark] SparkQA commented on pull request #29643: [SPARK-32638][SQL][FOLLOWUP] Move the plan rewriting methods to QueryPlan

SparkQA commented on pull request #29643: URL: https://github.com/apache/spark/pull/29643#issuecomment-686954059 **[Test build #128282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128282/testReport)** for PR 29643 at commit

[GitHub] [spark] SparkQA commented on pull request #29593: [SPARK-32753][SQL] Only copy tags to node with no tags

SparkQA commented on pull request #29593: URL: https://github.com/apache/spark/pull/29593#issuecomment-686957908 **[Test build #128283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128283/testReport)** for PR 29593 at commit

  1   2   3   4   5   >